2020-03-11T22:22:01.653 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:22:01.654 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.RUJMXG.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848258 2020-03-11T22:22:01.674 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:22:01.733 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848258 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848258' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.16497 2020-03-11T22:22:01.733 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:22:01.737 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:22:01.738 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:22:01.741 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:22:01.791 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:22:01.791 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:22:01.791 INFO:teuthology.packaging:ref: None 2020-03-11T22:22:01.791 INFO:teuthology.packaging:tag: None 2020-03-11T22:22:01.792 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:22:01.792 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:22:01.792 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:22:02.050 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:22:02.051 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:22:02.061 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:22:02.061 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:22:02.076 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:22:02.076 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:22:02.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 1)... 2020-03-11T22:22:12.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:22:22.165 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:22:32.176 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:22:42.194 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:22:52.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:02.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:12.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:22.264 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:32.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:42.300 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:23:52.316 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:24:02.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 1)... 2020-03-11T22:24:12.354 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 1)... 2020-03-11T22:24:22.365 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 1)... 2020-03-11T22:24:32.382 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 1)... 2020-03-11T22:24:42.401 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848258', 'machine_type': 'smithi'} 2020-03-11T22:24:42.415 DEBUG:teuthology.lock.ops:locked ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:24:42.442 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:24:42.642 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 8.1 2020-03-11T22:24:42.917 INFO:teuthology.orchestra.console:Power off smithi142 2020-03-11T22:24:42.917 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:24:42.939 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:46.960 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:51.074 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:55.186 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:55.399 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2020-03-11T22:24:55.500 INFO:teuthology.orchestra.console:Power on smithi142 2020-03-11T22:24:55.500 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:24:55.522 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:24:59.554 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:25:03.671 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:25:03.880 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2020-03-11T22:25:03.981 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2020-03-11T22:27:34.865 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:43.930 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:52.986 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:02.042 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:11.102 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:20.190 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:29.271 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:35.275 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:35.584 INFO:teuthology.orchestra.run.smithi142:> hostname 2020-03-11T22:29:35.868 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2020-03-11T22:29:35.869 INFO:teuthology.orchestra.run.smithi142:> grep smithi142.front.sepia.ceph.com /etc/hosts 2020-03-11T22:29:35.923 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2020-03-11T22:29:35.924 INFO:teuthology.orchestra.run.smithi142:> sudo hostname smithi142 2020-03-11T22:29:36.043 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/smithi142.front.sepia.ceph.com/smithi142/g' /etc/hosts 2020-03-11T22:29:36.073 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/172.21.15.142/172.21.15.142/g' /etc/hosts 2020-03-11T22:29:36.141 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/smithi142.front.sepia.ceph.com/smithi142/g' /etc/hostname 2020-03-11T22:29:36.208 INFO:teuthology.orchestra.run.smithi142:> sudo hostname smithi142 2020-03-11T22:29:36.275 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2020-03-11T22:29:36.277 DEBUG:teuthology.parallel:result is None 2020-03-11T22:29:36.380 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2020-03-11T22:29:36.381 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:29:36.401 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:29:36.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:29:36.402 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:36.508 INFO:teuthology.orchestra.run.smithi142:> true 2020-03-11T22:29:36.668 INFO:teuthology.orchestra.run.smithi142:> uname -m 2020-03-11T22:29:36.723 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2020-03-11T22:29:36.723 INFO:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2020-03-11T22:29:36.777 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Linux" 2020-03-11T22:29:36.777 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8 (Core)" 2020-03-11T22:29:36.777 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2020-03-11T22:29:36.777 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:29:36.777 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8" 2020-03-11T22:29:36.778 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:29:36.778 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:29:36.778 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2020-03-11T22:29:36.778 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:29:36.778 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:29:36.778 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:29:36.778 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:36.779 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:29:36.779 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:29:36.779 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:29:36.779 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:29:36.779 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:36.780 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2020-03-11T22:29:36.802 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:29:36.821 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmsMkaYji/ReUTMF518Z5W1QLnoCSMJRF7iaISaBvMWrb84q8manRPylQ8LGtiXxaOV1N4oJ9hBX8TDSVXtmLtociOuP4sj0Dml3VMyfU/vQ0dzjD6kGCE9oPTsFVImB4mSrCsADfcQlFye7OmFzbdQ2Oi6TJ/eumcrNz5J0K7/BqEQRUkasSQFvn5ZdOcBPURc9fWbNfdACriO015d9c/07/CRYOKqDeOYArucxVFfq8VcS9L/IRSAowGsOXCxpwJvTQT3j7jkN02VfrFKMGCT5Z8yvlivlGk58Ot792xRvdPRRGunUz4xsQhVjA29nSLQVXqZu9lHB49wkmlezDUAV/xhMGQkHLvnRrRfM/0MSIXAKZ8IcykJXF/mqdPvjSs7owMzwvB+hMOqxZqPEoyk2ZVqAyFFmsq8FUyg9rxjkX1X7y7RS+GeJCOA/zoj+wMFZVwXFY3PcoN4mQ3CnLDbI+tTTJnNRQNrDaTLGeLkCUk3utqcJDId47kUih0n9U= 2020-03-11T22:29:36.821 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:29:36.864 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:29:36.870 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:29:36.883 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:29:36.893 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:29:36.909 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:24:42.406534', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi142.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/w3sZSEGFlADTi4YYDfLe9UVzaTxzxMd+bwG/YKkqCgEzsCLmfEogxJt6W+PIeuE9MbHYK1qogLXkdURdvTd8=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848258'} 2020-03-11T22:29:36.909 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:29:36.916 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:29:36.916 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:29:36.953 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:29:36.954 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:29:36.961 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:29:36.961 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:29:36.963 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:29:37.035 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:29:37.044 INFO:teuthology.orchestra.run.smithi142:> uname -m 2020-03-11T22:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2020-03-11T22:29:37.200 INFO:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2020-03-11T22:29:37.253 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Linux" 2020-03-11T22:29:37.254 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8 (Core)" 2020-03-11T22:29:37.254 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2020-03-11T22:29:37.254 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:29:37.254 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8" 2020-03-11T22:29:37.254 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:29:37.254 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:29:37.255 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2020-03-11T22:29:37.255 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:29:37.255 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:29:37.255 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:29:37.255 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:37.255 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:29:37.255 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:29:37.256 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:29:37.256 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:29:37.256 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:37.256 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2020-03-11T22:29:37.270 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:29:37.304 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:29:37.322 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:29:37.323 INFO:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:29:37.338 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:29:37.348 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:29:37.348 INFO:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:29:37.407 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:29:37.441 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:29:37.459 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:29:37.459 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:29:37.460 INFO:teuthology.orchestra.run.smithi142:> uname -r 2020-03-11T22:29:37.475 INFO:teuthology.orchestra.run.smithi142.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:29:37.475 INFO:teuthology.task.kernel:Running kernel on smithi142: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:29:37.476 INFO:teuthology.orchestra.run.smithi142:> sudo yum install -y kernel 2020-03-11T22:29:39.998 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1 day, 7:03:46 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:29:40.364 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:29:40.364 INFO:teuthology.orchestra.run.smithi142.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:29:40.522 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:29:40.523 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:29:40.524 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:29:40.656 INFO:teuthology.orchestra.run.smithi142:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:29:41.103 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1 day, 7:03:48 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:29:41.462 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:29:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:29:41.463 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2020-03-11T22:29:41.463 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:29:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:Reinstalling: 2020-03-11T22:29:41.464 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:29:41.464 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:29:41.464 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:29:41.464 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:29:41.464 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:41.465 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 3.1 M 2020-03-11T22:29:41.465 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 0 2020-03-11T22:29:41.466 INFO:teuthology.orchestra.run.smithi142.stderr:Operation aborted. 2020-03-11T22:29:41.553 INFO:teuthology.orchestra.run.smithi142.stdout:Is this ok [y/N]: 2020-03-11T22:29:41.553 INFO:teuthology.orchestra.run.smithi142:> sudo yum reinstall -y kernel || true 2020-03-11T22:29:41.938 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 1 day, 7:03:48 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:29:42.297 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:29:42.298 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:29:42.298 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2020-03-11T22:29:42.298 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:29:42.298 INFO:teuthology.orchestra.run.smithi142.stdout:Reinstalling: 2020-03-11T22:29:42.298 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:29:42.298 INFO:teuthology.orchestra.run.smithi142.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:29:42.298 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:42.299 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:29:42.299 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:29:42.299 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:42.299 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 3.1 M 2020-03-11T22:29:42.300 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 0 2020-03-11T22:29:42.300 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:29:42.371 INFO:teuthology.orchestra.run.smithi142.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 25 MB/s | 1.5 MB 00:00 2020-03-11T22:29:42.384 INFO:teuthology.orchestra.run.smithi142.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2020-03-11T22:29:42.385 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:29:42.386 INFO:teuthology.orchestra.run.smithi142.stdout:Total 35 MB/s | 3.1 MB 00:00 2020-03-11T22:29:42.409 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:29:42.586 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:29:42.586 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:29:42.663 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:29:42.664 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:29:42.799 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:29:42.922 INFO:teuthology.orchestra.run.smithi142.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:29:43.057 INFO:teuthology.orchestra.run.smithi142.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:29:43.057 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:29:43.149 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:29:43.149 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:29:43.373 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:29:43.374 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:29:43.374 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:29:43.374 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:29:43.602 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:29:43.602 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:43.602 INFO:teuthology.orchestra.run.smithi142.stdout:Reinstalled: 2020-03-11T22:29:43.602 INFO:teuthology.orchestra.run.smithi142.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:29:43.603 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:43.603 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:29:43.712 INFO:teuthology.orchestra.run.smithi142:> rpm -q kernel --last | head -n 1 2020-03-11T22:29:43.802 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:29:42 PM UTC 2020-03-11T22:29:43.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:29:43.803 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-11T22:29:43.804 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-11T22:29:43.804 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-11T22:29:43.804 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-11T22:29:43.804 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2020-03-11T22:29:43.804 INFO:teuthology.orchestra.run.smithi142:> rpm -q kernel --last | head -n 1 2020-03-11T22:29:43.845 INFO:teuthology.orchestra.run.smithi142.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:29:42 PM UTC 2020-03-11T22:29:43.847 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:29:43.847 INFO:teuthology.orchestra.run.smithi142:> sudo rpm -qi grub2-tools 2020-03-11T22:29:43.917 INFO:teuthology.orchestra.run.smithi142.stdout:Name : grub2-tools 2020-03-11T22:29:43.918 INFO:teuthology.orchestra.run.smithi142.stdout:Epoch : 1 2020-03-11T22:29:43.918 INFO:teuthology.orchestra.run.smithi142.stdout:Version : 2.02 2020-03-11T22:29:43.918 INFO:teuthology.orchestra.run.smithi142.stdout:Release : 78.el8 2020-03-11T22:29:43.918 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: x86_64 2020-03-11T22:29:43.918 INFO:teuthology.orchestra.run.smithi142.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-11T22:29:43.918 INFO:teuthology.orchestra.run.smithi142.stdout:Group : System Environment/Base 2020-03-11T22:29:43.918 INFO:teuthology.orchestra.run.smithi142.stdout:Size : 9192691 2020-03-11T22:29:43.919 INFO:teuthology.orchestra.run.smithi142.stdout:License : GPLv3+ 2020-03-11T22:29:43.919 INFO:teuthology.orchestra.run.smithi142.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-11T22:29:43.919 INFO:teuthology.orchestra.run.smithi142.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-11T22:29:43.919 INFO:teuthology.orchestra.run.smithi142.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-11T22:29:43.919 INFO:teuthology.orchestra.run.smithi142.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-11T22:29:43.919 INFO:teuthology.orchestra.run.smithi142.stdout:Relocations : (not relocatable) 2020-03-11T22:29:43.919 INFO:teuthology.orchestra.run.smithi142.stdout:Packager : CentOS BuildSystem 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout:Vendor : CentOS 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout:Summary : Support tools for GRUB. 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout:Description : 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-11T22:29:43.920 INFO:teuthology.orchestra.run.smithi142.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-11T22:29:43.921 INFO:teuthology.orchestra.run.smithi142.stdout:hardware devices. 2020-03-11T22:29:43.921 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:29:43.921 INFO:teuthology.orchestra.run.smithi142.stdout:This subpackage provides tools for support of all platforms. 2020-03-11T22:29:43.922 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-11T22:29:43.922 INFO:teuthology.task.kernel:Updating grub on smithi142 to boot 4.18.0-147.el8.x86_64 2020-03-11T22:29:43.922 INFO:teuthology.orchestra.run.smithi142:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-11T22:29:44.871 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2020-03-11T22:29:46.698 INFO:teuthology.orchestra.run.smithi142.stderr:done 2020-03-11T22:29:46.700 INFO:teuthology.orchestra.run.smithi142:> mktemp 2020-03-11T22:29:46.730 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.Kbu0KnaGb1 2020-03-11T22:29:46.730 INFO:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Kbu0KnaGb1 2020-03-11T22:29:46.805 INFO:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.Kbu0KnaGb1 2020-03-11T22:29:46.959 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.Kbu0KnaGb1 is 4KB 2020-03-11T22:29:47.012 INFO:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.Kbu0KnaGb1 2020-03-11T22:29:47.028 INFO:teuthology.orchestra.run.smithi142:> sudo /bin/ls /boot/loader/entries || true 2020-03-11T22:29:47.093 INFO:teuthology.orchestra.run.smithi142.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-11T22:29:47.093 INFO:teuthology.orchestra.run.smithi142.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-11T22:29:47.093 INFO:teuthology.orchestra.run.smithi142.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-11T22:29:47.094 INFO:teuthology.orchestra.run.smithi142:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-11T22:29:47.254 INFO:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2020-03-11T22:30:17.286 INFO:teuthology.misc:Re-opening connections... 2020-03-11T22:30:17.286 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:30:17.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:17.287 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:35.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2020-03-11T22:30:35.750 DEBUG:teuthology.misc:waited 18.4642219543 2020-03-11T22:30:36.750 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:30:36.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:36.751 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:39.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2020-03-11T22:30:39.814 DEBUG:teuthology.misc:waited 22.528083086 2020-03-11T22:30:40.814 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:30:40.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:40.815 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:42.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2020-03-11T22:30:42.886 DEBUG:teuthology.misc:waited 25.6001229286 2020-03-11T22:30:43.887 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:30:43.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:43.887 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:45.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2020-03-11T22:30:45.958 DEBUG:teuthology.misc:waited 28.6721200943 2020-03-11T22:30:46.959 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:30:46.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:46.960 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:30:49.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2020-03-11T22:30:49.030 DEBUG:teuthology.misc:waited 31.7442729473 2020-03-11T22:30:50.030 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:30:50.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:30:50.031 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:31:50.033 DEBUG:teuthology.orchestra.remote:timed out 2020-03-11T22:31:50.033 DEBUG:teuthology.misc:waited 92.7474329472 2020-03-11T22:31:51.034 INFO:teuthology.misc:trying to connect to ubuntu@smithi142.front.sepia.ceph.com 2020-03-11T22:31:51.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:31:51.035 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:31:51.353 INFO:teuthology.orchestra.run.smithi142:> true 2020-03-11T22:31:51.648 DEBUG:teuthology.misc:waited 94.3617660999 2020-03-11T22:31:52.648 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-11T22:31:52.649 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-11T22:31:52.649 INFO:teuthology.orchestra.run.smithi142:> uname -r 2020-03-11T22:31:52.665 INFO:teuthology.orchestra.run.smithi142.stdout:4.18.0-147.el8.x86_64 2020-03-11T22:31:52.666 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-11T22:31:52.666 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-11T22:31:53.667 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:31:53.667 INFO:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:31:53.791 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2020-03-11T22:31:53.803 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:31:53.810 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:31:53.810 INFO:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:31:53.826 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:31:53.835 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:31:53.847 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:31:53.847 INFO:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:31:53.914 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:31:53.923 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:31:53.924 INFO:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:31:53.955 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:31:53.969 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:31:53.970 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:31:53.982 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:31:53.982 INFO:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:31:54.040 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:31:54.054 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:31:54.054 INFO:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:31:54.094 INFO:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:31:54.179 INFO:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:31:54.234 INFO:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:31:54.307 INFO:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:31:54.358 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:31:54.443 INFO:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2020-03-11T22:31:54.553 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:31:54.640 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:31:54.648 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:31:54.648 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:31:54.724 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:31:54.738 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:31:54.739 INFO:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2020-03-11T22:31:54.805 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: [ OK ] 2020-03-11T22:31:54.806 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:31:54.806 INFO:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2020-03-11T22:31:54.828 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2020-03-11T22:31:54.828 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi142.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:31:54.829 INFO:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:31:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:31:54.897 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:31:54.897 INFO:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:31:54.973 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:31:54.985 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-03-11T22:31:54.985 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:31:55.187 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:31:55.194 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:31:56.012 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:31:58.205 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:32:01.443 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:32:01.480 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:01 +0000 (0:00:00.141) 0:00:00.141 ******* 2020-03-11T22:32:04.060 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi142.front.sepia.ceph.com] 2020-03-11T22:32:04.146 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:04 +0000 (0:00:02.666) 0:00:02.807 ******* 2020-03-11T22:32:04.473 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:32:04.474 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:32:04.509 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:04 +0000 (0:00:00.362) 0:00:03.170 ******* 2020-03-11T22:32:05.102 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-11T22:32:05.103 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:32:05.143 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:05 +0000 (0:00:00.633) 0:00:03.804 ******* 2020-03-11T22:32:05.168 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:32:05.170 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:05.205 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:05 +0000 (0:00:00.062) 0:00:03.866 ******* 2020-03-11T22:32:05.539 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-11T22:32:05.540 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:32:05.587 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:05 +0000 (0:00:00.381) 0:00:04.248 ******* 2020-03-11T22:32:06.612 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:32:06.613 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965925.6-108719639256318/source", "state": "file", "uid": 0} 2020-03-11T22:32:06.648 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:06 +0000 (0:00:01.061) 0:00:05.309 ******* 2020-03-11T22:32:07.196 INFO:teuthology.task.ansible.out: 2020-03-11T22:32:07.198 INFO:teuthology.task.ansible.out:TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCto 2020-03-11T22:32:07.199 INFO:teuthology.task.ansible.out:HDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:32:07.327 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-11T22:32:07.334 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:32:07.550 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:07 +0000 (0:00:00.901) 0:00:06.211 ******* 2020-03-11T22:32:08.230 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi142.front.sepia.ceph.com] 2020-03-11T22:32:08.326 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Wednesday 11 March 2020 22:32:08 +0000 (0:00:00.775) 0:00:06.987 ******* 2020-03-11T22:32:09.046 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi142.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.396649", "end": "2020-03-11 22:32:09.010396", "rc": 0, "start": "2020-03-11 22:32:08.613747", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:32:09.083 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.756) 0:00:07.744 ******* 2020-03-11T22:32:09.306 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:32:09.307 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.019166", "end": "2020-03-11 22:32:09.281513", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:32:09.262347", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:32:09.344 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.261) 0:00:08.005 ******* 2020-03-11T22:32:09.370 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:32:09.372 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:32:09.406 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.061) 0:00:08.066 ******* 2020-03-11T22:32:09.431 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:32:09.432 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.467 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.061) 0:00:08.128 ******* 2020-03-11T22:32:09.493 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:32:09.495 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.529 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.061) 0:00:08.190 ******* 2020-03-11T22:32:09.554 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:32:09.556 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.591 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.062) 0:00:08.252 ******* 2020-03-11T22:32:09.617 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:32:09.618 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.657 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.065) 0:00:08.318 ******* 2020-03-11T22:32:09.683 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:32:09.685 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.728 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:09 +0000 (0:00:00.070) 0:00:08.388 ******* 2020-03-11T22:32:09.774 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:32:09.775 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.776 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.797 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.799 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.800 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.810 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.811 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.822 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.833 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.844 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.845 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.856 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.857 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.869 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.869 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.881 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.881 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.894 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.895 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.906 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.906 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.918 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.928 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.929 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.940 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.941 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.951 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.962 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.963 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.974 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.975 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.985 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.996 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:09.997 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.008 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.019 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.019 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.030 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.031 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.042 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.042 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.053 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.054 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.065 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.065 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.076 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.087 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.087 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.098 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.099 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.110 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.110 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.121 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.121 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.132 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.133 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.144 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.144 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.155 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.156 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.166 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.167 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.178 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.179 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.190 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.191 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.201 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.202 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.213 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.214 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.224 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.225 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.236 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.237 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.248 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.249 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.260 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.261 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.273 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.273 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.284 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.284 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.296 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.296 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.308 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.309 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.320 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.320 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.331 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.332 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.343 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.343 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.354 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.355 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.365 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.366 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.377 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.377 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.389 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.389 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.400 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.401 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.412 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.413 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.424 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.424 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.435 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.436 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.447 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.447 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.458 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.509 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.781) 0:00:09.170 ******* 2020-03-11T22:32:10.535 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:32:10.573 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.063) 0:00:09.234 ******* 2020-03-11T22:32:10.598 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-03-11T22:32:10.600 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.637 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.064) 0:00:09.298 ******* 2020-03-11T22:32:10.664 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-03-11T22:32:10.666 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.701 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.064) 0:00:09.362 ******* 2020-03-11T22:32:10.727 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-03-11T22:32:10.728 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.763 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.061) 0:00:09.424 ******* 2020-03-11T22:32:10.788 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-11T22:32:10.790 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.824 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.061) 0:00:09.485 ******* 2020-03-11T22:32:10.851 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-03-11T22:32:10.852 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.886 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:10 +0000 (0:00:00.061) 0:00:09.547 ******* 2020-03-11T22:32:10.922 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-03-11T22:32:10.924 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.934 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.934 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.945 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.956 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.958 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.969 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.977 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.989 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.990 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:10.991 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.001 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.002 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.013 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.024 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.025 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.036 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.036 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.047 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.047 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.058 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.059 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.069 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.080 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.081 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.082 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.092 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.093 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.103 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.104 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.115 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.115 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.126 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.126 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.137 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.138 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.148 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.149 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.160 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.170 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.171 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:11.213 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:11 +0000 (0:00:00.327) 0:00:09.874 ******* 2020-03-11T22:32:11.978 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-11T22:32:11.980 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:32:12.043 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.829) 0:00:10.704 ******* 2020-03-11T22:32:12.068 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:32:12.070 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => { "msg": "Host smithi142.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:32:12.111 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.068) 0:00:10.772 ******* 2020-03-11T22:32:12.148 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-11T22:32:12.149 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:32:12.197 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.083) 0:00:10.855 ******* 2020-03-11T22:32:12.225 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-11T22:32:12.227 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:12.270 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.075) 0:00:10.930 ******* 2020-03-11T22:32:12.553 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:32:12.555 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.082198", "end": "2020-03-11 22:32:12.522940", "rc": 0, "start": "2020-03-11 22:32:12.440742", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:32:12.592 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.322) 0:00:11.253 ******* 2020-03-11T22:32:12.618 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-11T22:32:12.619 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:12.656 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.064) 0:00:11.317 ******* 2020-03-11T22:32:12.681 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:32:12.683 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:12.719 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.063) 0:00:11.380 ******* 2020-03-11T22:32:12.746 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:32:12.748 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:12.784 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.064) 0:00:11.445 ******* 2020-03-11T22:32:12.810 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:32:12.812 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:12.848 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.063) 0:00:11.509 ******* 2020-03-11T22:32:12.874 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-11T22:32:12.875 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:12.912 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.064) 0:00:11.573 ******* 2020-03-11T22:32:12.937 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:32:12.938 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:12.973 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:12 +0000 (0:00:00.061) 0:00:11.634 ******* 2020-03-11T22:32:13.008 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:32:13.010 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:32:13.046 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.073) 0:00:11.707 ******* 2020-03-11T22:32:13.072 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-11T22:32:13.082 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.110 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.063) 0:00:11.771 ******* 2020-03-11T22:32:13.136 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-11T22:32:13.138 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.175 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.064) 0:00:11.836 ******* 2020-03-11T22:32:13.200 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:32:13.202 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:32:13.237 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.062) 0:00:11.898 ******* 2020-03-11T22:32:13.263 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:32:13.264 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.302 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.064) 0:00:11.963 ******* 2020-03-11T22:32:13.328 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:32:13.329 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:32:13.370 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.068) 0:00:12.031 ******* 2020-03-11T22:32:13.396 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.433 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.063) 0:00:12.094 ******* 2020-03-11T22:32:13.459 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:32:13.461 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.495 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.061) 0:00:12.156 ******* 2020-03-11T22:32:13.520 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:32:13.522 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.558 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.063) 0:00:12.219 ******* 2020-03-11T22:32:13.583 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:32:13.585 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.622 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.063) 0:00:12.283 ******* 2020-03-11T22:32:13.648 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:32:13.649 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:32:13.685 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.062) 0:00:12.346 ******* 2020-03-11T22:32:13.721 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:32:13.723 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.758 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.073) 0:00:12.419 ******* 2020-03-11T22:32:13.794 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:32:13.796 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.832 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.073) 0:00:12.493 ******* 2020-03-11T22:32:13.868 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-11T22:32:13.870 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:13.918 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:13 +0000 (0:00:00.085) 0:00:12.579 ******* 2020-03-11T22:32:13.954 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-11T22:32:13.955 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:14.013 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.094) 0:00:12.673 ******* 2020-03-11T22:32:14.049 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-03-11T22:32:14.051 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:14.085 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.072) 0:00:12.746 ******* 2020-03-11T22:32:14.122 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-03-11T22:32:14.123 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:14.171 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.085) 0:00:12.831 ******* 2020-03-11T22:32:14.198 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-03-11T22:32:14.199 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:32:14.237 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.063) 0:00:12.895 ******* 2020-03-11T22:32:14.261 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:32:14.298 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.063) 0:00:12.959 ******* 2020-03-11T22:32:14.335 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:32:14.336 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:14.372 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.073) 0:00:13.033 ******* 2020-03-11T22:32:14.596 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-03-11T22:32:14.598 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:32:14.633 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:14 +0000 (0:00:00.261) 0:00:13.294 ******* 2020-03-11T22:32:15.154 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-03-11T22:32:15.155 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:32:15.611 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:32:15.671 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:15 +0000 (0:00:01.037) 0:00:14.332 ******* 2020-03-11T22:32:16.165 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi142.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:32:16.623 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:32:16.663 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.992) 0:00:15.324 ******* 2020-03-11T22:32:16.700 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:32:16.701 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:16.738 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.075) 0:00:15.399 ******* 2020-03-11T22:32:16.765 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:32:16.766 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:16.803 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.064) 0:00:15.464 ******* 2020-03-11T22:32:16.829 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:32:16.830 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:16.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.063) 0:00:15.528 ******* 2020-03-11T22:32:16.894 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:32:16.895 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:16.933 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.065) 0:00:15.594 ******* 2020-03-11T22:32:16.959 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:32:16.960 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:16.997 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:16 +0000 (0:00:00.063) 0:00:15.658 ******* 2020-03-11T22:32:17.022 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:32:17.024 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:17.062 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:17 +0000 (0:00:00.065) 0:00:15.723 ******* 2020-03-11T22:32:17.088 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-11T22:32:17.090 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:17.127 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:17 +0000 (0:00:00.064) 0:00:15.787 ******* 2020-03-11T22:32:17.163 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:32:17.164 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:17.201 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:17 +0000 (0:00:00.074) 0:00:15.862 ******* 2020-03-11T22:32:17.227 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:32:17.228 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:17.265 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:17 +0000 (0:00:00.063) 0:00:15.926 ******* 2020-03-11T22:32:17.518 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-11T22:32:17.520 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.067753", "end": "2020-03-11 22:32:17.500309", "rc": 0, "start": "2020-03-11 22:32:17.432556", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:17.560 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:17 +0000 (0:00:00.294) 0:00:16.220 ******* 2020-03-11T22:32:20.759 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:32:20.760 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:32:20.798 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:20 +0000 (0:00:03.238) 0:00:19.459 ******* 2020-03-11T22:32:20.824 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-03-11T22:32:20.825 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:20.863 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:20 +0000 (0:00:00.064) 0:00:19.524 ******* 2020-03-11T22:32:20.890 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-11T22:32:20.891 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:20.927 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:20 +0000 (0:00:00.063) 0:00:19.588 ******* 2020-03-11T22:32:20.953 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:32:20.953 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:20.991 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:20 +0000 (0:00:00.063) 0:00:19.651 ******* 2020-03-11T22:32:21.017 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:32:21.018 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:21.054 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:21 +0000 (0:00:00.063) 0:00:19.715 ******* 2020-03-11T22:32:21.515 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-11T22:32:21.516 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:32:21.557 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:21 +0000 (0:00:00.502) 0:00:20.218 ******* 2020-03-11T22:32:22.466 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:32:22.467 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:32:22.514 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:22 +0000 (0:00:00.956) 0:00:21.175 ******* 2020-03-11T22:32:23.026 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:32:23.027 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:32:23.492 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:32:23.934 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:32:23.978 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:23 +0000 (0:00:01.463) 0:00:22.639 ******* 2020-03-11T22:32:24.169 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:32:24.169 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:32:24.214 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:24 +0000 (0:00:00.235) 0:00:22.874 ******* 2020-03-11T22:32:24.685 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:32:24.686 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:32:25.119 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:32:25.558 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:32:25.597 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:25 +0000 (0:00:01.383) 0:00:24.258 ******* 2020-03-11T22:32:25.623 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:32:25.624 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:25.661 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:25 +0000 (0:00:00.064) 0:00:24.322 ******* 2020-03-11T22:32:25.700 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:32:25.702 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:32:25.747 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:25 +0000 (0:00:00.085) 0:00:24.408 ******* 2020-03-11T22:32:25.943 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:32:25.944 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:32:25.983 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:25 +0000 (0:00:00.236) 0:00:24.644 ******* 2020-03-11T22:32:26.019 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:32:26.021 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:26.030 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:26.041 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:26.052 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:32:26.095 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:26 +0000 (0:00:00.111) 0:00:24.756 ******* 2020-03-11T22:32:26.994 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:32:26.994 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:32:27.038 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:27 +0000 (0:00:00.942) 0:00:25.699 ******* 2020-03-11T22:32:27.066 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-11T22:32:27.067 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:27.111 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:27 +0000 (0:00:00.072) 0:00:25.771 ******* 2020-03-11T22:32:27.572 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-11T22:32:27.574 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:32:27.609 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:27 +0000 (0:00:00.498) 0:00:26.270 ******* 2020-03-11T22:32:28.102 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:32:28.102 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:32:28.141 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:28 +0000 (0:00:00.532) 0:00:26.802 ******* 2020-03-11T22:32:28.167 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-11T22:32:28.168 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:28.204 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:28 +0000 (0:00:00.062) 0:00:26.864 ******* 2020-03-11T22:32:28.404 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:32:28.405 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:32:28.442 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:28 +0000 (0:00:00.238) 0:00:27.103 ******* 2020-03-11T22:32:28.698 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:32:28.699 INFO:teuthology.task.ansible.out:fatal: [smithi142.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:32:28.736 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:28 +0000 (0:00:00.293) 0:00:27.397 ******* 2020-03-11T22:32:28.772 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-11T22:32:28.804 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:32:28.812 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:28 +0000 (0:00:00.076) 0:00:27.473 ******* 2020-03-11T22:32:29.297 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:32:29.334 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:29 +0000 (0:00:00.522) 0:00:27.995 ******* 2020-03-11T22:32:30.068 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:32:30.070 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:31:17 UTC", "ActiveEnterTimestampMonotonic": "11261881", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice nss-lookup.target network.target systemd-tmpfiles-setup.service -.mount var-run.mount local-fs.target time-sync.target basic.target tmp.mount sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:31:17 UTC", "AssertTimestampMonotonic": "11261033", "Before": "multi-user.target plymouth-quit.service xdm.service shutdown.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:31:17 UTC", "ConditionTimestampMonotonic": "11261033", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-11T22:32:30.071 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2387", "ExecMainStartTimestamp": "Wed 2020-03-11 22:31:17 UTC", "ExecMainStartTimestampMonotonic": "11261852", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:31:17 UTC] ; stop_time=[n/a] ; pid=2387 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:31:17 UTC", "InactiveExitTimestampMonotonic": "11261881", "InvocationID": "e48868a67a814530b25f6a57a6787d5b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-11T22:32:30.073 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2387", "MemoryAccounting": "yes", "MemoryCurrent": "1675264", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-11T22:32:30.074 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:31:17 UTC", "StateChangeTimestampMonotonic": "11261881", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:31:17 UTC", "WatchdogTimestampMonotonic": "11261880", "WatchdogUSec": "0"}} 2020-03-11T22:32:30.107 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:30 +0000 (0:00:00.773) 0:00:28.768 ******* 2020-03-11T22:32:30.298 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:32:30.299 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001688", "end": "2020-03-11 22:32:30.275994", "rc": 0, "start": "2020-03-11 22:32:30.274306", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:32:30.335 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:30 +0000 (0:00:00.227) 0:00:28.996 ******* 2020-03-11T22:32:31.236 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-11T22:32:31.237 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:32:31.274 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:31 +0000 (0:00:00.939) 0:00:29.935 ******* 2020-03-11T22:32:32.174 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:32:32.175 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:32:32.218 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:32 +0000 (0:00:00.943) 0:00:30.878 ******* 2020-03-11T22:32:33.677 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:32:33.714 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:33 +0000 (0:00:01.496) 0:00:32.375 ******* 2020-03-11T22:32:50.207 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:32:50.208 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.293501", "end": "2020-03-11 22:32:50.176323", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:32:33.882822", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:32:50.243 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:50 +0000 (0:00:16.529) 0:00:48.904 ******* 2020-03-11T22:32:50.733 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-11T22:32:50.735 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:32:50.773 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:50 +0000 (0:00:00.529) 0:00:49.434 ******* 2020-03-11T22:32:50.985 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:32:50.986 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.008924", "end": "2020-03-11 22:32:50.957171", "rc": 0, "start": "2020-03-11 22:32:50.948247", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:51.024 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:51 +0000 (0:00:00.250) 0:00:49.685 ******* 2020-03-11T22:32:51.225 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:32:51.226 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.001988", "end": "2020-03-11 22:32:51.206934", "rc": 0, "start": "2020-03-11 22:32:51.204946", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:32:51.262 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:32:51 +0000 (0:00:00.238) 0:00:49.923 ******* 2020-03-11T22:33:07.595 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:33:07.596 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.142405", "end": "2020-03-11 22:33:07.572586", "rc": 0, "start": "2020-03-11 22:32:51.430181", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:07.636 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:07 +0000 (0:00:16.373) 0:01:06.297 ******* 2020-03-11T22:33:07.838 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:33:07.839 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:33:07.878 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:07 +0000 (0:00:00.242) 0:01:06.539 ******* 2020-03-11T22:33:07.914 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:33:07.915 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:33:07.951 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:07 +0000 (0:00:00.073) 0:01:06.612 ******* 2020-03-11T22:33:07.998 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:33:07.999 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:33:08.038 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.086) 0:01:06.699 ******* 2020-03-11T22:33:08.075 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:33:08.076 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:33:08.116 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.077) 0:01:06.777 ******* 2020-03-11T22:33:08.153 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:33:08.155 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:33:08.193 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.075) 0:01:06.854 ******* 2020-03-11T22:33:08.396 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:33:08.397 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:33:08.440 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.246) 0:01:07.101 ******* 2020-03-11T22:33:08.633 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:33:08.635 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:33:08.670 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.230) 0:01:07.331 ******* 2020-03-11T22:33:08.696 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.735 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.064) 0:01:07.396 ******* 2020-03-11T22:33:08.761 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:33:08.762 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:08.798 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:08 +0000 (0:00:00.063) 0:01:07.459 ******* 2020-03-11T22:33:09.014 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:33:09.015 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:33:09.052 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.254) 0:01:07.713 ******* 2020-03-11T22:33:09.731 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:33:09.732 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:33:09.769 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.716) 0:01:08.430 ******* 2020-03-11T22:33:09.805 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi142"}, "changed": false} 2020-03-11T22:33:09.843 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:09 +0000 (0:00:00.074) 0:01:08.504 ******* 2020-03-11T22:33:10.878 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi142.front.sepia.ceph.com", "ansible_hostname": "smithi142", "ansible_nodename": "smithi142"}, "changed": true, "name": "smithi142"} 2020-03-11T22:33:10.915 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:10 +0000 (0:00:01.072) 0:01:09.576 ******* 2020-03-11T22:33:11.387 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:33:11.423 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:11 +0000 (0:00:00.507) 0:01:10.084 ******* 2020-03-11T22:33:11.891 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:33:11.892 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:33:11.930 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:11 +0000 (0:00:00.506) 0:01:10.590 ******* 2020-03-11T22:33:12.524 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** ok: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:33:12.560 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:12 +0000 (0:00:00.630) 0:01:11.221 ******* 2020-03-11T22:33:13.033 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:33:13.075 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.515) 0:01:11.736 ******* 2020-03-11T22:33:13.281 INFO:teuthology.task.ansible.out: 2020-03-11T22:33:13.281 INFO:teuthology.task.ansible.out:TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:33:13.281 INFO:teuthology.task.ansible.out: 2020-03-11T22:33:13.283 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:33:13.322 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.246) 0:01:11.982 ******* 2020-03-11T22:33:13.522 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:33:13.524 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:33:13.559 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.237) 0:01:12.220 ******* 2020-03-11T22:33:13.780 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:33:13.782 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:33:13.816 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:13 +0000 (0:00:00.257) 0:01:12.477 ******* 2020-03-11T22:33:14.296 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:33:14.297 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:33:14.343 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:14 +0000 (0:00:00.527) 0:01:13.004 ******* 2020-03-11T22:33:14.821 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:33:14.822 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:33:14.861 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:14 +0000 (0:00:00.517) 0:01:13.521 ******* 2020-03-11T22:33:16.707 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:33:16.708 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.646365", "end": "2020-03-11 22:33:16.677602", "rc": 0, "start": "2020-03-11 22:33:15.031237", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:16.746 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:16 +0000 (0:00:01.885) 0:01:15.407 ******* 2020-03-11T22:33:17.256 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:33:17.257 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:33:17.710 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:33:18.175 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:33:18.213 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:18 +0000 (0:00:01.466) 0:01:16.873 ******* 2020-03-11T22:33:18.694 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-03-11T22:33:18.695 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:33:18.736 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:18 +0000 (0:00:00.523) 0:01:17.397 ******* 2020-03-11T22:33:19.245 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:33:19.246 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:33:19.802 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:33:20.271 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:33:20.314 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:20 +0000 (0:00:01.575) 0:01:18.972 ******* 2020-03-11T22:33:20.837 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** changed: [smithi142.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.311766", "end": "2020-03-11 22:33:20.807208", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:33:20.495442", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:33:22.850 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:20 +0000 (0:00:00.566) 0:01:19.539 ******* TASK [testnode : Install GPG] ************************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:33:20 +0000 (0:00:00.073) 0:01:19.613 ******* TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi142.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:33:21 +0000 (0:00:00.100) 0:01:19.713 ******* TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:33:21 +0000 (0:00:00.077) 0:01:19.791 ******* TASK [testnode : set_fact] ***************************************************** ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} Wednesday 11 March 2020 22:33:21 +0000 (0:00:00.091) 0:01:19.883 ******* TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:33:21 +0000 (0:00:00.085) 0:01:19.968 ******* TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": f 2020-03-11T22:33:22.851 INFO:teuthology.task.ansible.out:alse, "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:33:21 +0000 (0:00:00.071) 0:01:20.040 ******* TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi142.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006741", "end": "2020-03-11 22:33:21.567457", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:33:21.560716", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring Wednesday 11 March 2020 22:33:21 +0000 (0:00:00.263) 0:01:20.303 ******* TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:33:21 +0000 (0:00:00.082) 0:01:20.385 ******* 2020-03-11T22:33:24.107 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:33:24.107 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:33:24.144 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:24 +0000 (0:00:02.419) 0:01:22.805 ******* 2020-03-11T22:33:27.222 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:33:27.223 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:33:27.279 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:27 +0000 (0:00:03.134) 0:01:25.940 ******* 2020-03-11T22:33:32.389 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-11T22:33:32.390 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:04.884552", "end": "2020-03-11 22:33:32.359928", "rc": 0, "start": "2020-03-11 22:33:27.475376", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\""], "stdout": "CentOS-8 - AppStream 141 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 123 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 65 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 138 kB/s | 5.3 kB 00:00 \nlab-extras 1.3 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:02 ago on Wed 11 Mar 2020 10:33:28 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-11T22:33:32.391 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 141 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 123 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 51 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 65 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 138 kB/s | 5.3 kB 00:00 ", "lab-extras 1.3 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:02 ago on Wed 11 Mar 2020 10:33:28 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:33:32.435 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:32 +0000 (0:00:05.156) 0:01:31.096 ******* 2020-03-11T22:33:39.940 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:33:39.941 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:33:39.979 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:39 +0000 (0:00:07.543) 0:01:38.640 ******* 2020-03-11T22:33:40.015 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:33:40.017 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:40.055 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:40 +0000 (0:00:00.075) 0:01:38.716 ******* 2020-03-11T22:33:40.990 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:33:40.992 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:33:41.032 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:41 +0000 (0:00:00.976) 0:01:39.692 ******* 2020-03-11T22:33:41.069 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:33:41.071 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:41.128 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:41 +0000 (0:00:00.096) 0:01:39.789 ******* 2020-03-11T22:33:41.447 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-11T22:33:41.449 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target dbus.socket system.slice polkit.service basic.target dbus.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ip6tables.service ebtables.service iptables.service ipset.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-11T22:33:41.450 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", 2020-03-11T22:33:41.452 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-11T22:33:41.453 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:33:41.489 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:41 +0000 (0:00:00.361) 0:01:40.150 ******* 2020-03-11T22:33:41.794 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-11T22:33:41.795 INFO:teuthology.task.ansible.out:fatal: [smithi142.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:33:41.830 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:41 +0000 (0:00:00.340) 0:01:40.491 ******* 2020-03-11T22:33:42.641 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:33:42.642 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:33:42.689 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:42 +0000 (0:00:00.858) 0:01:41.350 ******* 2020-03-11T22:33:42.717 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:33:42.717 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:42.754 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:42 +0000 (0:00:00.064) 0:01:41.415 ******* 2020-03-11T22:33:42.779 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:33:42.816 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:42 +0000 (0:00:00.062) 0:01:41.477 ******* 2020-03-11T22:33:42.842 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:33:42.844 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:42.882 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:42 +0000 (0:00:00.064) 0:01:41.542 ******* 2020-03-11T22:33:42.955 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 11 March 2020 22:33:42 +0000 (0:00:00.064) 0:01:41.606 ******* 2020-03-11T22:33:42.982 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-11T22:33:42.984 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.021 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.075) 0:01:41.682 ******* 2020-03-11T22:33:43.047 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-11T22:33:43.049 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.090 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.069) 0:01:41.751 ******* 2020-03-11T22:33:43.116 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:33:43.153 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.062) 0:01:41.814 ******* 2020-03-11T22:33:43.179 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:33:43.180 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.216 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.063) 0:01:41.877 ******* 2020-03-11T22:33:43.243 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:33:43.245 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.280 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.064) 0:01:41.941 ******* 2020-03-11T22:33:43.308 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:33:43.310 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.345 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.064) 0:01:42.006 ******* 2020-03-11T22:33:43.372 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:33:43.373 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.407 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.062) 0:01:42.068 ******* 2020-03-11T22:33:43.434 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-03-11T22:33:43.435 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.475 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.067) 0:01:42.136 ******* 2020-03-11T22:33:43.504 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:33:43.506 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.548 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.072) 0:01:42.208 ******* 2020-03-11T22:33:43.574 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:33:43.575 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.615 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.066) 0:01:42.275 ******* 2020-03-11T22:33:43.641 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-03-11T22:33:43.643 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.682 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.066) 0:01:42.343 ******* 2020-03-11T22:33:43.711 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:33:43.712 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.747 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.065) 0:01:42.408 ******* 2020-03-11T22:33:43.783 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:33:43.785 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.820 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.072) 0:01:42.480 ******* 2020-03-11T22:33:43.845 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:33:43.847 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.881 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.061) 0:01:42.542 ******* 2020-03-11T22:33:43.907 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-11T22:33:43.908 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:43.944 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:43 +0000 (0:00:00.062) 0:01:42.605 ******* 2020-03-11T22:33:43.969 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:33:43.971 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.005 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.061) 0:01:42.666 ******* 2020-03-11T22:33:44.032 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:33:44.033 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.069 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.063) 0:01:42.730 ******* 2020-03-11T22:33:44.094 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:33:44.096 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.131 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.061) 0:01:42.792 ******* 2020-03-11T22:33:44.157 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:33:44.158 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.194 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.063) 0:01:42.855 ******* 2020-03-11T22:33:44.230 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:33:44.232 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.266 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.072) 0:01:42.927 ******* 2020-03-11T22:33:44.307 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:33:44.309 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.343 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.076) 0:01:43.003 ******* 2020-03-11T22:33:44.378 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:33:44.380 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.416 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.073) 0:01:43.077 ******* 2020-03-11T22:33:44.442 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:33:44.443 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.479 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.062) 0:01:43.140 ******* 2020-03-11T22:33:44.506 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:33:44.507 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.556 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.076) 0:01:43.216 ******* 2020-03-11T22:33:44.583 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:33:44.585 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.642 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.086) 0:01:43.303 ******* 2020-03-11T22:33:44.670 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:33:44.672 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.709 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.066) 0:01:43.369 ******* 2020-03-11T22:33:44.734 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:33:44.736 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.771 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.062) 0:01:43.432 ******* 2020-03-11T22:33:44.797 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:33:44.799 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.835 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.063) 0:01:43.496 ******* 2020-03-11T22:33:44.871 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:33:44.872 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:44.908 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:44 +0000 (0:00:00.073) 0:01:43.569 ******* 2020-03-11T22:33:45.140 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:33:45.141 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:33:45.189 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:45 +0000 (0:00:00.280) 0:01:43.850 ******* 2020-03-11T22:33:45.432 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:33:45.434 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:33:45.470 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:45 +0000 (0:00:00.280) 0:01:44.130 ******* 2020-03-11T22:33:45.695 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:33:45.697 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:33:45.763 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:45 +0000 (0:00:00.292) 0:01:44.423 ******* 2020-03-11T22:33:45.790 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:33:45.792 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:45.826 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:45 +0000 (0:00:00.063) 0:01:44.486 ******* 2020-03-11T22:33:45.851 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:33:45.853 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:45.892 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:45 +0000 (0:00:00.066) 0:01:44.553 ******* 2020-03-11T22:33:45.919 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:33:45.920 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:45.957 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:45 +0000 (0:00:00.065) 0:01:44.618 ******* 2020-03-11T22:33:45.993 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:33:46.004 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.036 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.079) 0:01:44.697 ******* 2020-03-11T22:33:46.062 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:33:46.064 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.108 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.071) 0:01:44.768 ******* 2020-03-11T22:33:46.133 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:33:46.135 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.177 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.069) 0:01:44.838 ******* 2020-03-11T22:33:46.213 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:33:46.215 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.251 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.073) 0:01:44.912 ******* 2020-03-11T22:33:46.277 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:33:46.279 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.316 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.065) 0:01:44.977 ******* 2020-03-11T22:33:46.343 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:33:46.344 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.405 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.088) 0:01:45.065 ******* 2020-03-11T22:33:46.433 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-11T22:33:46.434 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.475 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.068) 0:01:45.134 ******* 2020-03-11T22:33:46.499 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:33:46.501 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.537 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.063) 0:01:45.197 ******* 2020-03-11T22:33:46.572 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:33:46.574 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.610 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.073) 0:01:45.271 ******* 2020-03-11T22:33:46.647 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:33:46.654 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.690 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.080) 0:01:45.351 ******* 2020-03-11T22:33:46.720 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:33:46.721 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.761 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.070) 0:01:45.422 ******* 2020-03-11T22:33:46.786 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:33:46.788 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.826 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.064) 0:01:45.486 ******* 2020-03-11T22:33:46.851 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-11T22:33:46.853 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.900 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.074) 0:01:45.561 ******* 2020-03-11T22:33:46.928 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-03-11T22:33:46.930 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:46.966 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:46 +0000 (0:00:00.064) 0:01:45.626 ******* 2020-03-11T22:33:46.992 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:33:46.993 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.032 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.066) 0:01:45.692 ******* 2020-03-11T22:33:47.060 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-11T22:33:47.064 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.105 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.073) 0:01:45.766 ******* 2020-03-11T22:33:47.131 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:33:47.133 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.186 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.080) 0:01:45.847 ******* 2020-03-11T22:33:47.214 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-03-11T22:33:47.216 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.255 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.069) 0:01:45.916 ******* 2020-03-11T22:33:47.283 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-11T22:33:47.285 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.323 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.068) 0:01:45.984 ******* 2020-03-11T22:33:47.349 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:33:47.351 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:33:47.388 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.064) 0:01:46.049 ******* 2020-03-11T22:33:47.435 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:33:47.436 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.456 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:33:47.457 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.468 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.479 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.479 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.490 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.533 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.144) 0:01:46.194 ******* 2020-03-11T22:33:47.745 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-11T22:33:47.746 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009659", "end": "2020-03-11 22:33:47.717883", "rc": 0, "start": "2020-03-11 22:33:47.708224", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:33:47.786 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.253) 0:01:46.447 ******* 2020-03-11T22:33:47.832 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:33:47.834 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.873 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.087) 0:01:46.534 ******* 2020-03-11T22:33:47.899 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:33:47.901 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:47.938 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:47 +0000 (0:00:00.064) 0:01:46.599 ******* 2020-03-11T22:33:47.964 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:33:47.966 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:48.003 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:48 +0000 (0:00:00.064) 0:01:46.663 ******* 2020-03-11T22:33:48.028 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-11T22:33:48.030 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:48.065 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:48 +0000 (0:00:00.062) 0:01:46.726 ******* 2020-03-11T22:33:48.998 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:33:49.000 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:33:49.040 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:49 +0000 (0:00:00.974) 0:01:47.701 ******* 2020-03-11T22:33:49.087 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:33:49.089 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item={u'block_used': 12964175, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130933, u'size_available': 930328301568, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130933, "block_size": 4096, "block_total": 240095108, "block_used": 12964175, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328301568, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:33:49.125 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:49 +0000 (0:00:00.084) 0:01:47.786 ******* 2020-03-11T22:33:49.343 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:33:49.345 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005005", "end": "2020-03-11 22:33:49.316232", "rc": 0, "start": "2020-03-11 22:33:49.311227", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:49.379 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:49 +0000 (0:00:00.254) 0:01:48.040 ******* 2020-03-11T22:33:49.427 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:33:49.427 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'block_used': 12964175, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130933, u'size_available': 930328301568, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130933, "block_size": 4096, "block_total": 240095108, "block_used": 12964175, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328301568, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:49.465 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:49 +0000 (0:00:00.086) 0:01:48.126 ******* 2020-03-11T22:33:49.491 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:33:49.529 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:49 +0000 (0:00:00.063) 0:01:48.190 ******* 2020-03-11T22:33:49.700 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:33:49.701 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ', u'wwn-0x5000c5009292426c'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFVQ', u'wwn': u'0x5000c5009292426c', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1', u'wwn-0x5000c5009292426c-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:33:49.703 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:51.204 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-plvvRN-i8rO-R8pM-unYk-HXsf-29NP-gzojK9', u'nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN', u'nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.317446", "end": "2020-03-11 22:33:51.173597", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-plvvRN-i8rO-R8pM-unYk-HXsf-29NP-gzojK9", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:49.856151", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-11T22:33:51.205 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:33:52.459 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMkRLjxToNfo5tByh73KfoAR4jt0IcbNaL'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.059390", "end": "2020-03-11 22:33:52.429989", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMkRLjxToNfo5tByh73KfoAR4jt0IcbNaL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:51.370599", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:33:53.701 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykM8tus7BE6IdnfN0JfGBweNYMphNEQBrXy'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.038823", "end": "2020-03-11 22:33:53.665882", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykM8tus7BE6IdnfN0JfGBweNYMphNEQBrXy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:52.627059", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:33:54.930 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMqKZ0LpvrdiSbYLjQeNNTSVZJL8JBzUHQ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.038841", "end": "2020-03-11 22:33:54.899334", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMqKZ0LpvrdiSbYLjQeNNTSVZJL8JBzUHQ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:53.860493", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:33:56.175 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMCukGYaiWPEISoIAABA8f9PCDOVp0fzpu'], u'uuids': [u'9d4e1f43-88d4-4a19-be85-c1fefd2bff0e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.048052", "end": "2020-03-11 22:33:56.140699", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMCukGYaiWPEISoIAABA8f9PCDOVp0fzpu"], "labels": [], "masters": [], "uuids": ["9d4e1f43-88d4-4a19-be85-c1fefd2bff0e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:55.092647", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:33:57.414 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMzzUaKIIeVOSwCch7GoO4JkKcXNju9qWX'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.038677", "end": "2020-03-11 22:33:57.382490", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMzzUaKIIeVOSwCch7GoO4JkKcXNju9qWX"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:56.343813", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:33:57.468 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:57 +0000 (0:00:07.938) 0:01:56.129 ******* 2020-03-11T22:33:57.629 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-03-11T22:33:57.631 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ', u'wwn-0x5000c5009292426c'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XFVQ', u'wwn': u'0x5000c5009292426c', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1', u'wwn-0x5000c5009292426c-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ", "wwn-0x5000c5009292426c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFVQ-part1", "wwn-0x5000c5009292426c-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFVQ", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:33:57.633 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009292426c"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:33:57.859 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-plvvRN-i8rO-R8pM-unYk-HXsf-29NP-gzojK9', u'nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN', u'nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037276", "end": "2020-03-11 22:33:57.827465", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-plvvRN-i8rO-R8pM-unYk-HXsf-29NP-gzojK9", "nvme-INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN", "nvme-nvme.8086-43564654363330333030395934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:57.790189", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-11T22:33:57.861 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0346838 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0346838 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:58.180 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMkRLjxToNfo5tByh73KfoAR4jt0IcbNaL'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.139690", "end": "2020-03-11 22:33:58.153276", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMkRLjxToNfo5tByh73KfoAR4jt0IcbNaL"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:58.013586", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137152 s, 841 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137152 s, 841 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:58.516 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykM8tus7BE6IdnfN0JfGBweNYMphNEQBrXy'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.141767", "end": "2020-03-11 22:33:58.482017", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykM8tus7BE6IdnfN0JfGBweNYMphNEQBrXy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:58.340250", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139159 s, 829 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139159 s, 829 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:58.847 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMqKZ0LpvrdiSbYLjQeNNTSVZJL8JBzUHQ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.145963", "end": "2020-03-11 22:33:58.813879", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMqKZ0LpvrdiSbYLjQeNNTSVZJL8JBzUHQ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:58.667916", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143337 s, 805 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143337 s, 805 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:59.167 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMCukGYaiWPEISoIAABA8f9PCDOVp0fzpu'], u'uuids': [u'9d4e1f43-88d4-4a19-be85-c1fefd2bff0e']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.139571", "end": "2020-03-11 22:33:59.135886", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMCukGYaiWPEISoIAABA8f9PCDOVp0fzpu"], "labels": [], "masters": [], "uuids": ["9d4e1f43-88d4-4a19-be85-c1fefd2bff0e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:58.996315", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137052 s, 842 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137052 s, 842 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:59.616 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMzzUaKIIeVOSwCch7GoO4JkKcXNju9qWX'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.138309", "end": "2020-03-11 22:33:59.467189", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-OlQhCCq0d2IFMgs5NNZKME1dDSPMVykMzzUaKIIeVOSwCch7GoO4JkKcXNju9qWX"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:33:59.328880", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.135794 s, 849 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.135794 s, 849 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:59.659 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:59 +0000 (0:00:02.190) 0:01:58.320 ******* 2020-03-11T22:33:59.931 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:33:59.933 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.063513", "end": "2020-03-11 22:33:59.894156", "rc": 0, "start": "2020-03-11 22:33:59.830643", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:33:59.973 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:33:59 +0000 (0:00:00.314) 0:01:58.634 ******* 2020-03-11T22:34:00.215 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-11T22:34:00.216 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043541", "end": "2020-03-11 22:34:00.193105", "rc": 0, "start": "2020-03-11 22:34:00.149564", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:34:00.254 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.280) 0:01:58.914 ******* 2020-03-11T22:34:00.280 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:34:00.320 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.066) 0:01:58.981 ******* 2020-03-11T22:34:00.346 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:34:00.347 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:00.387 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.067) 0:01:59.048 ******* 2020-03-11T22:34:00.413 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:34:00.416 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:00.452 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.064) 0:01:59.112 ******* 2020-03-11T22:34:00.478 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:34:00.479 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:00.523 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.071) 0:01:59.184 ******* 2020-03-11T22:34:00.549 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:34:00.550 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:00.587 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.064) 0:01:59.248 ******* 2020-03-11T22:34:00.780 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:34:00.781 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:34:00.821 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.233) 0:01:59.482 ******* 2020-03-11T22:34:00.859 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:34:00.860 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={u'block_used': 12964175, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130933, u'size_available': 930328301568, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130933, "block_size": 4096, "block_total": 240095108, "block_used": 12964175, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328301568, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:00.906 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.084) 0:01:59.566 ******* 2020-03-11T22:34:00.932 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-11T22:34:00.933 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:00.975 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:00 +0000 (0:00:00.068) 0:01:59.635 ******* 2020-03-11T22:34:01.002 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:34:01.003 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:01.043 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:01 +0000 (0:00:00.068) 0:01:59.704 ******* 2020-03-11T22:34:01.072 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:34:01.073 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:01.121 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:01 +0000 (0:00:00.078) 0:01:59.782 ******* 2020-03-11T22:34:01.147 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:34:01.149 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:01.190 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:01 +0000 (0:00:00.068) 0:01:59.851 ******* 2020-03-11T22:34:01.216 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-11T22:34:01.218 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:01.259 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:01 +0000 (0:00:00.068) 0:01:59.920 ******* 2020-03-11T22:34:01.285 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-11T22:34:01.286 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:01.329 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:01 +0000 (0:00:00.069) 0:01:59.989 ******* 2020-03-11T22:34:01.957 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-11T22:34:01.958 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:34:01.996 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:01 +0000 (0:00:00.667) 0:02:00.657 ******* 2020-03-11T22:34:02.538 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:34:02.540 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:34:02.882 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:34:03.224 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:34:03.547 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:34:03.869 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:34:03.910 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:03 +0000 (0:00:01.913) 0:02:02.571 ******* 2020-03-11T22:34:04.113 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:34:04.115 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:34:04.161 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:04 +0000 (0:00:00.250) 0:02:02.822 ******* 2020-03-11T22:34:04.211 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:34:04.213 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:04.400 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:34:04.588 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:34:04.775 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:34:04.956 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:34:05.039 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:05 +0000 (0:00:00.877) 0:02:03.699 ******* 2020-03-11T22:34:05.243 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:34:05.244 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:34:05.279 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:05 +0000 (0:00:00.240) 0:02:03.940 ******* 2020-03-11T22:34:05.315 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-11T22:34:05.316 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:34:05.359 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:05 +0000 (0:00:00.080) 0:02:04.020 ******* 2020-03-11T22:34:06.804 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:34:06.805 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:34:06.847 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:06 +0000 (0:00:01.487) 0:02:05.508 ******* 2020-03-11T22:34:07.327 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:34:07.328 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:34:07.366 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.518) 0:02:06.027 ******* 2020-03-11T22:34:07.392 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-11T22:34:07.393 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.433 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.066) 0:02:06.094 ******* 2020-03-11T22:34:07.460 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:34:07.461 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.498 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.064) 0:02:06.159 ******* 2020-03-11T22:34:07.523 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-11T22:34:07.525 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.561 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.063) 0:02:06.222 ******* 2020-03-11T22:34:07.587 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:34:07.588 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.624 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.063) 0:02:06.285 ******* 2020-03-11T22:34:07.654 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.692 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.067) 0:02:06.353 ******* 2020-03-11T22:34:07.718 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:34:07.719 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.759 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.067) 0:02:06.420 ******* 2020-03-11T22:34:07.785 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:34:07.787 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.823 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.063) 0:02:06.483 ******* 2020-03-11T22:34:07.848 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:34:07.850 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.886 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.063) 0:02:06.546 ******* 2020-03-11T22:34:07.911 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:34:07.913 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:07.947 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:07 +0000 (0:00:00.061) 0:02:06.608 ******* 2020-03-11T22:34:07.974 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:34:07.976 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:08.011 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:08 +0000 (0:00:00.063) 0:02:06.672 ******* 2020-03-11T22:34:08.037 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-11T22:34:08.039 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:08.076 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:08 +0000 (0:00:00.065) 0:02:06.737 ******* 2020-03-11T22:34:08.555 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-11T22:34:08.557 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:34:08.593 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:08 +0000 (0:00:00.517) 0:02:07.254 ******* 2020-03-11T22:34:08.898 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:34:08.900 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:31:17 UTC", "ActiveEnterTimestampMonotonic": "11033486", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service sysinit.target system.slice systemd-journald.socket sntp.service -.mount ntpdate.service tmp.mount ntpd.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:31:16 UTC", "AssertTimestampMonotonic": "9398296", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:31:16 UTC", "ConditionTimestampMonotonic": "9398263", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": 2020-03-11T22:34:08.902 INFO:teuthology.task.ansible.out:"NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1499", "ExecMainStartTimestamp": "Wed 2020-03-11 22:31:17 UTC", "ExecMainStartTimestampMonotonic": "10985798", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:31:16 UTC", "InactiveExitTimestampMonotonic": "9420344", "InvocationID": "47a484ab998a4cb8aec8bfc0752d57b5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216" 2020-03-11T22:34:08.903 INFO:teuthology.task.ansible.out:, "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127436", "LimitNPROCSoft": "127436", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127436", "LimitSIGPENDINGSoft": "127436", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1499", "MemoryAccounting": "yes", "MemoryCurrent": "3510272", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardErro 2020-03-11T22:34:08.905 INFO:teuthology.task.ansible.out:r": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:31:17 UTC", "StateChangeTimestampMonotonic": "11033486", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:31:17 UTC", "WatchdogTimestampMonotonic": "10985811", "WatchdogUSec": "0"}} 2020-03-11T22:34:08.938 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:08 +0000 (0:00:00.344) 0:02:07.599 ******* 2020-03-11T22:34:08.964 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-11T22:34:08.965 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:09.000 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:08 +0000 (0:00:00.061) 0:02:07.661 ******* 2020-03-11T22:34:09.025 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-11T22:34:09.027 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:09.062 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:09 +0000 (0:00:00.062) 0:02:07.723 ******* 2020-03-11T22:34:09.088 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-11T22:34:09.089 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:09.124 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:09 +0000 (0:00:00.062) 0:02:07.785 ******* 2020-03-11T22:34:09.325 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-11T22:34:09.325 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:34:09.363 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:09 +0000 (0:00:00.237) 0:02:08.023 ******* 2020-03-11T22:34:09.830 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-11T22:34:09.832 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:34:09.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:09 +0000 (0:00:00.504) 0:02:08.528 ******* 2020-03-11T22:34:11.039 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:34:11.040 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:34:11.080 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:01.212) 0:02:09.741 ******* 2020-03-11T22:34:11.106 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:34:11.108 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:11.147 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:00.067) 0:02:09.808 ******* 2020-03-11T22:34:11.185 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-11T22:34:11.221 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:00.073) 0:02:09.882 ******* 2020-03-11T22:34:11.424 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:34:11.425 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:34:11.459 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:00.238) 0:02:10.120 ******* 2020-03-11T22:34:11.485 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:34:11.487 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:11.530 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:00.070) 0:02:10.191 ******* 2020-03-11T22:34:11.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:34:11.570 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:34:11.605 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:00.075) 0:02:10.266 ******* 2020-03-11T22:34:11.631 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-11T22:34:11.633 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:11.669 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:00.064) 0:02:10.330 ******* 2020-03-11T22:34:11.870 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-11T22:34:11.871 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:34:11.906 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:11 +0000 (0:00:00.236) 0:02:10.567 ******* 2020-03-11T22:34:12.098 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-11T22:34:12.099 INFO:teuthology.task.ansible.out:changed: [smithi142.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:34:12.136 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.229) 0:02:10.797 ******* 2020-03-11T22:34:12.162 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.200 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.063) 0:02:10.861 ******* 2020-03-11T22:34:12.201 INFO:teuthology.task.ansible.out: 2020-03-11T22:34:12.236 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-11T22:34:12.238 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi142.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.248 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.249 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.250 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.287 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.087) 0:02:10.948 ******* 2020-03-11T22:34:12.324 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:34:12.325 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.326 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.337 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.338 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.350 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.389 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.101) 0:02:11.050 ******* 2020-03-11T22:34:12.415 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:34:12.416 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.457 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.067) 0:02:11.118 ******* 2020-03-11T22:34:12.484 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:34:12.485 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.527 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.069) 0:02:11.187 ******* 2020-03-11T22:34:12.555 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:34:12.557 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.608 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.081) 0:02:11.269 ******* 2020-03-11T22:34:12.634 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-11T22:34:12.636 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.673 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.064) 0:02:11.334 ******* 2020-03-11T22:34:12.698 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:34:12.700 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.740 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.066) 0:02:11.401 ******* 2020-03-11T22:34:12.765 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:34:12.769 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.805 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.064) 0:02:11.466 ******* 2020-03-11T22:34:12.831 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:34:12.833 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.871 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.066) 0:02:11.532 ******* 2020-03-11T22:34:12.908 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:34:12.909 INFO:teuthology.task.ansible.out:skipping: [smithi142.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:34:12.948 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:12 +0000 (0:00:00.075) 0:02:11.607 ******* 2020-03-11T22:34:13.209 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:34:13.211 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005077", "end": "2020-03-11 22:34:13.163392", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:34:13.158315", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:34:13.405 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002948", "end": "2020-03-11 22:34:13.364866", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:34:13.361918", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:34:13.591 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002753", "end": "2020-03-11 22:34:13.555467", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:34:13.552714", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:34:13.807 INFO:teuthology.task.ansible.out:ok: [smithi142.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.037583", "end": "2020-03-11 22:34:13.774630", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:34:13.737047", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:34:13.846 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:13 +0000 (0:00:00.899) 0:02:12.507 ******* 2020-03-11T22:34:13.872 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:34:13.908 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:34:13 +0000 (0:00:00.061) 0:02:12.569 ******* 2020-03-11T22:34:14.123 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi142.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:34:14.230 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-03-11T22:34:14.232 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-03-11T22:34:14.234 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-03-11T22:34:14.236 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi142.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Wednesday 11 March 2020 22:34:14 +0000 (0:00:00.326) 0:02:12.895 ******* =============================================================================== common : nrpe - Remove SELinux policy package -------------------------- 16.53s common : nrpe - Load SELinux policy package ---------------------------- 16.37s testnode : Zap all non-root disks --------------------------------------- 7.94s testnode : Install packages --------------------------------------------- 7.54s testnode : Base group workaround for BZ1782899 -------------------------- 5.16s common : Install Kerberos Packages (RedHat) ----------------------------- 3.24s testnode : Ensure ceph dependency packages are not present. ------------- 3.13s Gathering Facts --------------------------------------------------------- 2.67s testnode : Ensure ceph packages are not present. ------------------------ 2.42s testnode : Blow away lingering OSD data and FSIDs ----------------------- 2.19s testnode : Create logical volume(s) ------------------------------------- 1.91s testnode : Rebuild rpmdb ------------------------------------------------ 1.89s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.58s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.50s testnode : Create xfs filesystem on /dev/vg_nvme/lv_5 ------------------- 1.49s testnode : Configure local mirrorlists ---------------------------------- 1.47s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.46s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.38s testnode : Install packages via pip ------------------------------------- 1.21s testnode : Set the system's h 2020-03-11T22:34:14.237 INFO:teuthology.task.ansible.out:ostname ------------------------------------ 1.07s 2020-03-11T22:34:14.526 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2020-03-11T22:34:14.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:34:14.529 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi142.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:34:14.596 INFO:teuthology.orchestra.run.smithi142:> true 2020-03-11T22:34:14.682 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:34:14.696 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:34:14.696 INFO:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:34:14.756 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:34:14.774 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:34:14.815 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: ntpd: command not found 2020-03-11T22:34:14.829 INFO:teuthology.orchestra.run.smithi142.stdout:506 Cannot talk to daemon 2020-03-11T22:34:14.846 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:34:14.863 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:34:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:bash: ntpq: command not found 2020-03-11T22:34:14.926 INFO:teuthology.orchestra.run.smithi142.stdout:210 Number of sources = 4 2020-03-11T22:34:14.926 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:34:14.926 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2020-03-11T22:34:14.926 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:34:14.926 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:34:14.926 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:34:14.926 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:34:14.927 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:34:14.939 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:34:14.939 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:34:14.939 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:34:14.940 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:34:14.954 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:34:14.955 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:34:14.955 INFO:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2020-03-11T22:34:15.001 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS 2020-03-11T22:34:15.002 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:34:15.003 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:34:15.003 INFO:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2020-03-11T22:34:15.078 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS 2020-03-11T22:34:15.079 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:34:15.079 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:34:15.332 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:34:15.333 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:34:15.519 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:34:15.519 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:34:15.564 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:34:15.564 INFO:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:34:15.632 INFO:teuthology.orchestra.run.smithi142:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:34:15.712 INFO:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:34:15.779 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2020-03-11T22:34:15.780 INFO:teuthology.orchestra.run.smithi142:> sudo yum clean all 2020-03-11T22:34:16.187 INFO:teuthology.orchestra.run.smithi142.stdout:239 files removed 2020-03-11T22:34:16.216 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw 2020-03-11T22:34:16.934 INFO:teuthology.orchestra.run.smithi142.stdout:ceph packages for x86_64 184 kB/s | 69 kB 00:00 2020-03-11T22:34:17.249 INFO:teuthology.orchestra.run.smithi142.stdout:ceph noarch packages 47 kB/s | 14 kB 00:00 2020-03-11T22:34:17.552 INFO:teuthology.orchestra.run.smithi142.stdout:ceph source packages 5.0 kB/s | 1.4 kB 00:00 2020-03-11T22:34:17.707 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-8 - AppStream 48 MB/s | 6.5 MB 00:00 2020-03-11T22:34:18.833 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-8 - Base 41 MB/s | 5.0 MB 00:00 2020-03-11T22:34:19.605 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS-8 - Extras 32 kB/s | 2.1 kB 00:00 2020-03-11T22:34:19.743 INFO:teuthology.orchestra.run.smithi142.stdout:Copr repo for ceph-el8 owned by ktdreyer 406 kB/s | 38 kB 00:00 2020-03-11T22:34:20.187 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 15 MB/s | 6.0 MB 00:00 2020-03-11T22:34:21.568 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 5.4 MB/s | 30 kB 00:00 2020-03-11T22:34:22.533 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:34:22.535 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:34:22.536 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2020-03-11T22:34:22.536 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:34:22.536 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:34:22.536 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:34:22.536 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrading: 2020-03-11T22:34:22.536 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:34:22.536 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:34:22.537 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:34:22.537 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:34:22.537 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:34:22.537 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:34:22.537 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:34:22.537 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:34:22.537 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:34:22.538 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:34:22.538 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:34:22.538 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:34:22.538 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:34:22.538 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:34:22.538 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:34:22.538 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:34:22.539 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:22.540 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:34:22.540 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:34:22.540 INFO:teuthology.orchestra.run.smithi142.stdout:Install 21 Packages 2020-03-11T22:34:22.540 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 2 Packages 2020-03-11T22:34:22.540 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:22.541 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 51 M 2020-03-11T22:34:22.541 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:34:23.201 INFO:teuthology.orchestra.run.smithi142.stdout:(1/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 8.3 MB/s | 5.5 MB 00:00 2020-03-11T22:34:23.234 INFO:teuthology.orchestra.run.smithi142.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 711 kB/s | 25 kB 00:00 2020-03-11T22:34:23.282 INFO:teuthology.orchestra.run.smithi142.stdout:(3/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 10 MB/s | 507 kB 00:00 2020-03-11T22:34:23.334 INFO:teuthology.orchestra.run.smithi142.stdout:(4/23): libradosstriper1-15.1.0-1941.g4acc112.e 7.1 MB/s | 377 kB 00:00 2020-03-11T22:34:23.606 INFO:teuthology.orchestra.run.smithi142.stdout:(5/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 9.2 MB/s | 9.8 MB 00:01 2020-03-11T22:34:23.664 INFO:teuthology.orchestra.run.smithi142.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 696 kB/s | 40 kB 00:00 2020-03-11T22:34:23.722 INFO:teuthology.orchestra.run.smithi142.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 845 kB/s | 49 kB 00:00 2020-03-11T22:34:23.782 INFO:teuthology.orchestra.run.smithi142.stdout:(8/23): librgw2-15.1.0-1941.g4acc112.el8.x86_64 14 MB/s | 6.2 MB 00:00 2020-03-11T22:34:23.784 INFO:teuthology.orchestra.run.smithi142.stdout:(9/23): python3-cephfs-15.1.0-1941.g4acc112.el8 2.6 MB/s | 162 kB 00:00 2020-03-11T22:34:23.828 INFO:teuthology.orchestra.run.smithi142.stdout:(10/23): python3-rados-15.1.0-1941.g4acc112.el8 7.5 MB/s | 352 kB 00:00 2020-03-11T22:34:23.845 INFO:teuthology.orchestra.run.smithi142.stdout:(11/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 4.9 MB/s | 309 kB 00:00 2020-03-11T22:34:23.860 INFO:teuthology.orchestra.run.smithi142.stdout:(12/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 3.5 MB/s | 113 kB 00:00 2020-03-11T22:34:23.879 INFO:teuthology.orchestra.run.smithi142.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2020-03-11T22:34:23.881 INFO:teuthology.orchestra.run.smithi142.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 34 MB/s | 44 kB 00:00 2020-03-11T22:34:23.882 INFO:teuthology.orchestra.run.smithi142.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 33 MB/s | 47 kB 00:00 2020-03-11T22:34:23.892 INFO:teuthology.orchestra.run.smithi142.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.1 MB/s | 259 kB 00:00 2020-03-11T22:34:23.954 INFO:teuthology.orchestra.run.smithi142.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 954 kB/s | 59 kB 00:00 2020-03-11T22:34:23.984 INFO:teuthology.orchestra.run.smithi142.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.4 MB/s | 75 kB 00:00 2020-03-11T22:34:23.990 INFO:teuthology.orchestra.run.smithi142.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 32 MB/s | 179 kB 00:00 2020-03-11T22:34:23.995 INFO:teuthology.orchestra.run.smithi142.stdout:(20/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.7 MB/s | 315 kB 00:00 2020-03-11T22:34:24.261 INFO:teuthology.orchestra.run.smithi142.stdout:(21/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 7.2 MB/s | 1.9 MB 00:00 2020-03-11T22:34:24.273 INFO:teuthology.orchestra.run.smithi142.stdout:(22/23): librados2-15.1.0-1941.g4acc112.el8.x86 13 MB/s | 3.6 MB 00:00 2020-03-11T22:34:24.967 INFO:teuthology.orchestra.run.smithi142.stdout:(23/23): ceph-common-15.1.0-1941.g4acc112.el8.x 8.5 MB/s | 21 MB 00:02 2020-03-11T22:34:24.970 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:24.971 INFO:teuthology.orchestra.run.smithi142.stdout:Total 21 MB/s | 51 MB 00:02 2020-03-11T22:34:24.976 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:34:25.018 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:34:25.019 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:34:25.358 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:34:25.358 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:34:26.359 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:34:26.552 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:34:26.726 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:34:27.115 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:34:27.343 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:34:27.428 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:34:27.627 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:34:27.776 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:34:28.007 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:34:28.193 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:34:28.264 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:34:28.442 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:34:28.584 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:34:29.202 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:34:29.367 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:34:29.422 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:34:29.560 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:34:29.730 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:34:29.955 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:34:30.151 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:34:30.318 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:34:30.396 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:34:30.589 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:34:30.800 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:34:30.942 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:34:31.124 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:34:31.268 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:34:34.741 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:34:34.934 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:34:35.611 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:34:35.907 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:34:35.998 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:34:36.142 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:34:52.450 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:34:52.450 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /sys 2020-03-11T22:34:52.450 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /proc 2020-03-11T22:34:52.450 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /dev 2020-03-11T22:34:52.451 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /run 2020-03-11T22:34:52.451 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /mnt 2020-03-11T22:34:52.451 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /var/tmp 2020-03-11T22:34:52.451 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /home 2020-03-11T22:34:52.451 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /tmp 2020-03-11T22:34:52.451 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /dev 2020-03-11T22:34:52.451 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:53.797 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:34:53.864 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:34:53.864 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:34:54.057 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:34:54.058 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:34:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:34:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:34:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:34:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:34:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:34:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:34:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:34:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:34:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:34:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:34:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:34:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:34:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:34:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:34:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:34:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:34:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:34:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:34:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:34:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:34:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:34:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:34:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:34:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:34:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:34:55.375 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:34:55.375 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:55.375 INFO:teuthology.orchestra.run.smithi142.stdout:Upgraded: 2020-03-11T22:34:55.375 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.375 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:55.376 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:34:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.376 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.377 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.377 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.377 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.377 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.377 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.377 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.377 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.378 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:34:55.378 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:34:55.378 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:34:55.378 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:34:55.378 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:34:55.378 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:34:55.378 INFO:teuthology.orchestra.run.smithi142.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:34:55.379 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:34:55.379 INFO:teuthology.orchestra.run.smithi142.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:34:55.379 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:55.379 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:34:55.647 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-test 2020-03-11T22:34:56.056 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:35 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:34:56.425 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:34:56.426 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:34:56.426 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2020-03-11T22:34:56.426 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:34:56.426 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:34:56.427 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:34:56.427 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:34:56.427 INFO:teuthology.orchestra.run.smithi142.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:34:56.427 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:34:56.427 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:34:56.427 INFO:teuthology.orchestra.run.smithi142.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:34:56.428 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:34:56.428 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:56.428 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:34:56.428 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:34:56.428 INFO:teuthology.orchestra.run.smithi142.stdout:Install 6 Packages 2020-03-11T22:34:56.428 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:34:56.429 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 43 M 2020-03-11T22:34:56.429 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 184 M 2020-03-11T22:34:56.429 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:34:56.462 INFO:teuthology.orchestra.run.smithi142.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.8 MB/s | 161 kB 00:00 2020-03-11T22:34:56.471 INFO:teuthology.orchestra.run.smithi142.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2020-03-11T22:34:56.472 INFO:teuthology.orchestra.run.smithi142.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 28 MB/s | 298 kB 00:00 2020-03-11T22:34:56.479 INFO:teuthology.orchestra.run.smithi142.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 31 MB/s | 249 kB 00:00 2020-03-11T22:34:56.535 INFO:teuthology.orchestra.run.smithi142.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.1 MB/s | 69 kB 00:00 2020-03-11T22:34:58.527 INFO:teuthology.orchestra.run.smithi142.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 20 MB/s | 42 MB 00:02 2020-03-11T22:34:58.530 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:34:58.530 INFO:teuthology.orchestra.run.smithi142.stdout:Total 20 MB/s | 43 MB 00:02 2020-03-11T22:34:58.535 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:34:58.555 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:34:58.556 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:34:58.817 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:34:58.817 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:34:59.239 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:34:59.974 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:35:00.159 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:35:00.351 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:35:00.521 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:35:00.618 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:35:04.614 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:35:04.834 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:35:05.054 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:35:05.054 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:35:05.055 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:35:05.055 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:35:05.055 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:35:05.055 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:35:06.120 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:35:06.120 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:06.120 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:06.120 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:35:06.120 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:35:06.121 INFO:teuthology.orchestra.run.smithi142.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:35:06.121 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:06.121 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:07.280 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph 2020-03-11T22:35:07.687 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:00:46 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:08.086 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:08.090 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:08.090 INFO:teuthology.orchestra.run.smithi142.stdout: Package ArchVersion Repository Size 2020-03-11T22:35:08.090 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:08.090 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:35:08.091 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:35:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:35:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:35:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:35:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:35:08.091 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:35:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:35:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:35:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:35:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:35:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:35:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:35:08.092 INFO:teuthology.orchestra.run.smithi142.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:35:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:35:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:35:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:35:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:35:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:35:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:35:08.093 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:35:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:35:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:35:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:35:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:35:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:35:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:35:08.094 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:35:08.095 INFO:teuthology.orchestra.run.smithi142.stdout:Installing weak dependencies: 2020-03-11T22:35:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:35:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:35:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:08.096 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:08.096 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:08.096 INFO:teuthology.orchestra.run.smithi142.stdout:Install 36 Packages 2020-03-11T22:35:08.096 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:08.097 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 29 M 2020-03-11T22:35:08.097 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 105 M 2020-03-11T22:35:08.097 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:08.247 INFO:teuthology.orchestra.run.smithi142.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 43 kB/s | 6.4 kB 00:00 2020-03-11T22:35:08.515 INFO:teuthology.orchestra.run.smithi142.stdout:(2/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 3.4 MB/s | 1.4 MB 00:00 2020-03-11T22:35:08.558 INFO:teuthology.orchestra.run.smithi142.stdout:(3/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 4.4 MB/s | 2.0 MB 00:00 2020-03-11T22:35:08.625 INFO:teuthology.orchestra.run.smithi142.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.4 MB/s | 165 kB 00:00 2020-03-11T22:35:08.656 INFO:teuthology.orchestra.run.smithi142.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.7 MB/s | 214 kB 00:00 2020-03-11T22:35:08.675 INFO:teuthology.orchestra.run.smithi142.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 84 MB/s | 1.5 MB 00:00 2020-03-11T22:35:08.678 INFO:teuthology.orchestra.run.smithi142.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 51 MB/s | 157 kB 00:00 2020-03-11T22:35:08.679 INFO:teuthology.orchestra.run.smithi142.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 29 MB/s | 39 kB 00:00 2020-03-11T22:35:08.681 INFO:teuthology.orchestra.run.smithi142.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 35 MB/s | 54 kB 00:00 2020-03-11T22:35:08.682 INFO:teuthology.orchestra.run.smithi142.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 20 MB/s | 27 kB 00:00 2020-03-11T22:35:08.689 INFO:teuthology.orchestra.run.smithi142.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 75 MB/s | 457 kB 00:00 2020-03-11T22:35:08.691 INFO:teuthology.orchestra.run.smithi142.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 63 MB/s | 195 kB 00:00 2020-03-11T22:35:08.693 INFO:teuthology.orchestra.run.smithi142.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 27 MB/s | 34 kB 00:00 2020-03-11T22:35:08.695 INFO:teuthology.orchestra.run.smithi142.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 51 MB/s | 123 kB 00:00 2020-03-11T22:35:08.699 INFO:teuthology.orchestra.run.smithi142.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 60 MB/s | 176 kB 00:00 2020-03-11T22:35:08.763 INFO:teuthology.orchestra.run.smithi142.stdout:(16/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_ 8.5 MB/s | 4.4 MB 00:00 2020-03-11T22:35:08.764 INFO:teuthology.orchestra.run.smithi142.stdout:(17/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.2 MB/s | 145 kB 00:00 2020-03-11T22:35:08.775 INFO:teuthology.orchestra.run.smithi142.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.1 MB/s | 11 kB 00:00 2020-03-11T22:35:08.784 INFO:teuthology.orchestra.run.smithi142.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.0 MB/s | 18 kB 00:00 2020-03-11T22:35:08.795 INFO:teuthology.orchestra.run.smithi142.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 5.2 MB/s | 49 kB 00:00 2020-03-11T22:35:08.803 INFO:teuthology.orchestra.run.smithi142.stdout:(21/36): python3-more-itertools-7.2.0-3.el8.noa 6.4 MB/s | 59 kB 00:00 2020-03-11T22:35:08.821 INFO:teuthology.orchestra.run.smithi142.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 15 MB/s | 283 kB 00:00 2020-03-11T22:35:08.830 INFO:teuthology.orchestra.run.smithi142.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-03-11T22:35:08.833 INFO:teuthology.orchestra.run.smithi142.stdout:(24/36): python3-cherrypy-18.4.0-1.el8.noarch.r 5.4 MB/s | 385 kB 00:00 2020-03-11T22:35:08.838 INFO:teuthology.orchestra.run.smithi142.stdout:(25/36): python3-simplegeneric-0.8.1-17.el8.noa 2.3 MB/s | 19 kB 00:00 2020-03-11T22:35:08.842 INFO:teuthology.orchestra.run.smithi142.stdout:(26/36): python3-singledispatch-3.4.0.3-18.el8. 2.9 MB/s | 24 kB 00:00 2020-03-11T22:35:08.846 INFO:teuthology.orchestra.run.smithi142.stdout:(27/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.6 MB/s | 30 kB 00:00 2020-03-11T22:35:08.850 INFO:teuthology.orchestra.run.smithi142.stdout:(28/36): python3-trustme-0.5.2-4.el8.noarch.rpm 3.2 MB/s | 27 kB 00:00 2020-03-11T22:35:08.855 INFO:teuthology.orchestra.run.smithi142.stdout:(29/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.8 MB/s | 23 kB 00:00 2020-03-11T22:35:08.925 INFO:teuthology.orchestra.run.smithi142.stdout:(30/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 582 kB/s | 44 kB 00:00 2020-03-11T22:35:08.944 INFO:teuthology.orchestra.run.smithi142.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 2.1 MB/s | 40 kB 00:00 2020-03-11T22:35:08.960 INFO:teuthology.orchestra.run.smithi142.stdout:(32/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2020-03-11T22:35:08.983 INFO:teuthology.orchestra.run.smithi142.stdout:(33/36): python3-waitress-1.2.1-2.el8.1.noarch. 6.1 MB/s | 241 kB 00:00 2020-03-11T22:35:08.986 INFO:teuthology.orchestra.run.smithi142.stdout:(34/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.5 MB/s | 251 kB 00:00 2020-03-11T22:35:09.002 INFO:teuthology.orchestra.run.smithi142.stdout:(35/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.3 MB/s | 85 kB 00:00 2020-03-11T22:35:09.360 INFO:teuthology.orchestra.run.smithi142.stdout:(36/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 20 MB/s | 17 MB 00:00 2020-03-11T22:35:09.365 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:09.365 INFO:teuthology.orchestra.run.smithi142.stdout:Total 23 MB/s | 29 MB 00:01 2020-03-11T22:35:09.385 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:09.423 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:09.423 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:09.771 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:09.771 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:10.991 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:11.206 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:35:11.416 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:35:11.681 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:35:11.853 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:35:12.010 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:35:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:35:12.375 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:35:12.507 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:35:12.662 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:35:12.830 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:35:12.975 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:35:13.156 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:35:13.359 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:35:13.587 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:35:13.869 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:35:14.082 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:35:14.329 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:35:14.546 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:35:14.796 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:35:15.022 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:35:15.347 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:35:15.592 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:35:15.791 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:35:15.980 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:35:16.174 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:35:16.391 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:35:16.609 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:35:16.828 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:35:17.159 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:35:17.423 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:35:17.766 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:35:17.925 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:35:19.345 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:35:19.518 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:35:19.911 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:35:20.061 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:35:20.253 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:35:20.443 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:35:20.462 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:35:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:35:20.912 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:35:20.913 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:35:20.913 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:35:20.913 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:35:20.913 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:35:20.913 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:35:20.913 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:35:20.914 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:35:20.914 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:35:20.914 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:35:20.914 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:35:20.914 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:35:20.915 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:35:20.915 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:35:20.915 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:35:20.915 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:35:20.916 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:35:20.916 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:35:20.916 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:35:20.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:35:20.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:35:20.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:35:20.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:35:20.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:35:20.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:35:20.917 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:35:20.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:35:20.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:35:20.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:35:20.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:35:20.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:35:20.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:35:20.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:35:20.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:35:20.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:35:20.919 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:35:21.637 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:35:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:21.638 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:35:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:35:21.638 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:35:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:35:21.639 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:35:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:35:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:35:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:35:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:35:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:35:21.640 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:35:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:35:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:35:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:35:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:35:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:35:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:35:21.641 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:35:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:35:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:35:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:35:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:35:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:35:21.642 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:35:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:35:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:35:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:35:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:35:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:35:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:35:21.643 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:35:21.644 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:21.644 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:21.907 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-base 2020-03-11T22:35:22.311 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:22.573 INFO:teuthology.orchestra.run.smithi142.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:22.680 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:22.681 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:35:22.681 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:22.847 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install cephadm 2020-03-11T22:35:23.240 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:23.604 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:23.605 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:23.605 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2020-03-11T22:35:23.605 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:23.606 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:23.606 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:35:23.606 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:23.606 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:23.606 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:23.606 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2020-03-11T22:35:23.606 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:23.607 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 36 k 2020-03-11T22:35:23.607 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 129 k 2020-03-11T22:35:23.607 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:23.794 INFO:teuthology.orchestra.run.smithi142.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 190 kB/s | 36 kB 00:00 2020-03-11T22:35:23.794 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:23.795 INFO:teuthology.orchestra.run.smithi142.stdout:Total 189 kB/s | 36 kB 00:00 2020-03-11T22:35:23.797 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:23.804 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:23.804 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:23.806 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:23.806 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:23.854 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:25.206 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:25.460 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:25.679 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:35:25.679 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:25.679 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:25.679 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:25.679 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:25.679 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:25.806 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:35:26.209 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:26.577 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:26.577 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:26.577 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2020-03-11T22:35:26.578 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:26.578 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:26.578 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:35:26.578 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:26.578 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:26.578 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:26.579 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2020-03-11T22:35:26.579 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:26.579 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 203 k 2020-03-11T22:35:26.579 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 743 k 2020-03-11T22:35:26.580 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:26.872 INFO:teuthology.orchestra.run.smithi142.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 691 kB/s | 203 kB 00:00 2020-03-11T22:35:26.873 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:26.873 INFO:teuthology.orchestra.run.smithi142.stdout:Total 688 kB/s | 203 kB 00:00 2020-03-11T22:35:26.875 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:26.887 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:26.887 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:26.933 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:26.933 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:27.068 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:27.217 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:35:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:35:28.038 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:35:28.039 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:28.039 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:28.039 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:35:28.039 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:28.039 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:28.314 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr 2020-03-11T22:35:28.720 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:28.989 INFO:teuthology.orchestra.run.smithi142.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:35:29.094 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:29.095 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:35:29.096 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:29.246 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:35:29.640 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:08 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:30.012 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:30.013 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:30.013 INFO:teuthology.orchestra.run.smithi142.stdout: Package ArchVersion Repository Size 2020-03-11T22:35:30.013 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:30.013 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:30.013 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:35:30.013 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:35:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:35:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:35:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:35:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:35:30.014 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:30.014 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:30.014 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:30.015 INFO:teuthology.orchestra.run.smithi142.stdout:Install 5 Packages 2020-03-11T22:35:30.015 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:30.015 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 4.2 M 2020-03-11T22:35:30.016 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 19 M 2020-03-11T22:35:30.016 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:30.068 INFO:teuthology.orchestra.run.smithi142.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 795 kB/s | 43 kB 00:00 2020-03-11T22:35:30.119 INFO:teuthology.orchestra.run.smithi142.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 679 kB/s | 34 kB 00:00 2020-03-11T22:35:30.143 INFO:teuthology.orchestra.run.smithi142.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 7.8 MB/s | 197 kB 00:00 2020-03-11T22:35:30.202 INFO:teuthology.orchestra.run.smithi142.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 119 kB/s | 22 kB 00:00 2020-03-11T22:35:30.442 INFO:teuthology.orchestra.run.smithi142.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 9.0 MB/s | 3.9 MB 00:00 2020-03-11T22:35:30.445 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:30.445 INFO:teuthology.orchestra.run.smithi142.stdout:Total 9.6 MB/s | 4.2 MB 00:00 2020-03-11T22:35:30.449 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:30.458 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:30.458 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:30.594 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:30.595 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:30.808 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:30.989 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:35:31.192 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:35:31.398 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:35:32.382 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:35:32.657 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:35:32.843 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:35:32.843 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:35:32.843 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:35:32.844 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:35:32.844 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:35:33.095 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:35:33.095 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:33.095 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:33.095 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:35:33.095 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:35:33.096 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:35:33.096 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:35:33.096 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:35:33.096 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:33.096 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:33.240 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:35:33.640 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:12 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:34.873 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:34.874 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:34.874 INFO:teuthology.orchestra.run.smithi142.stdout: Package ArchVersion Repository Size 2020-03-11T22:35:34.874 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:34.874 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:34.874 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:35:34.875 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:35:34.875 INFO:teuthology.orchestra.run.smithi142.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:35:34.875 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:35:34.875 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:35:34.875 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:35:34.875 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:35:34.875 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:35:34.876 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:35:34.876 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:34.876 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:34.876 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:34.876 INFO:teuthology.orchestra.run.smithi142.stdout:Install 8 Packages 2020-03-11T22:35:34.876 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:34.877 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 37 M 2020-03-11T22:35:34.877 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 205 M 2020-03-11T22:35:34.877 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:35.063 INFO:teuthology.orchestra.run.smithi142.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 23 MB/s | 4.3 MB 00:00 2020-03-11T22:35:35.150 INFO:teuthology.orchestra.run.smithi142.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2020-03-11T22:35:35.158 INFO:teuthology.orchestra.run.smithi142.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 31 MB/s | 225 kB 00:00 2020-03-11T22:35:35.190 INFO:teuthology.orchestra.run.smithi142.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 29 MB/s | 3.7 MB 00:00 2020-03-11T22:35:35.203 INFO:teuthology.orchestra.run.smithi142.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 50 MB/s | 638 kB 00:00 2020-03-11T22:35:35.208 INFO:teuthology.orchestra.run.smithi142.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 32 MB/s | 169 kB 00:00 2020-03-11T22:35:35.399 INFO:teuthology.orchestra.run.smithi142.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 60 MB/s | 15 MB 00:00 2020-03-11T22:35:38.933 INFO:teuthology.orchestra.run.smithi142.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 2.1 MB/s | 8.6 MB 00:04 2020-03-11T22:35:38.941 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:38.941 INFO:teuthology.orchestra.run.smithi142.stdout:Total 9.0 MB/s | 37 MB 00:04 2020-03-11T22:35:38.953 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:38.973 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:38.973 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:39.262 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:39.262 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:39.775 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:39.914 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:35:40.055 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:35:40.214 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:35:40.750 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:35:40.906 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:35:41.448 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:35:41.614 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:35:42.310 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:35:42.617 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:35:42.839 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:35:44.799 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:35:46.079 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:35:46.255 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:35:46.494 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:35:46.494 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:35:46.494 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:35:46.494 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:35:46.494 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:35:46.494 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:35:46.495 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:35:46.495 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:35:46.776 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:35:46.776 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:46.776 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:46.777 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:35:46.777 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:35:46.777 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:35:46.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:35:46.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:35:46.777 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:35:46.777 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:35:46.778 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:35:46.778 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:46.778 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:46.911 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:35:47.307 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:26 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:47.683 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:47.685 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:47.685 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2020-03-11T22:35:47.686 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:47.686 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:47.686 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:35:47.686 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-noarch 24 k 2020-03-11T22:35:47.686 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:35:47.686 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:35:47.686 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:35:47.687 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:35:47.687 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:35:47.687 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:35:47.687 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:35:47.687 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:35:47.687 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:35:47.687 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout:Install 13 Packages 2020-03-11T22:35:47.688 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:47.689 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 1.9 M 2020-03-11T22:35:47.689 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 23 M 2020-03-11T22:35:47.689 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:47.719 INFO:teuthology.orchestra.run.smithi142.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 888 kB/s | 27 kB 00:00 2020-03-11T22:35:47.722 INFO:teuthology.orchestra.run.smithi142.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 35 MB/s | 126 kB 00:00 2020-03-11T22:35:47.723 INFO:teuthology.orchestra.run.smithi142.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 586 kB/s | 20 kB 00:00 2020-03-11T22:35:47.730 INFO:teuthology.orchestra.run.smithi142.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 14 MB/s | 110 kB 00:00 2020-03-11T22:35:47.731 INFO:teuthology.orchestra.run.smithi142.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 19 MB/s | 155 kB 00:00 2020-03-11T22:35:47.732 INFO:teuthology.orchestra.run.smithi142.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 27 MB/s | 43 kB 00:00 2020-03-11T22:35:47.789 INFO:teuthology.orchestra.run.smithi142.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 288 kB/s | 16 kB 00:00 2020-03-11T22:35:47.794 INFO:teuthology.orchestra.run.smithi142.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 523 kB/s | 33 kB 00:00 2020-03-11T22:35:47.865 INFO:teuthology.orchestra.run.smithi142.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.0 MB/s | 81 kB 00:00 2020-03-11T22:35:47.875 INFO:teuthology.orchestra.run.smithi142.stdout:(10/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 130 kB/s | 24 kB 00:00 2020-03-11T22:35:47.893 INFO:teuthology.orchestra.run.smithi142.stdout:(11/13): python3-rsa-3.4.2-12.el8.noarch.rpm 2.6 MB/s | 73 kB 00:00 2020-03-11T22:35:47.917 INFO:teuthology.orchestra.run.smithi142.stdout:(12/13): python3-kubernetes-10.0.1-2.el8.noarch 9.1 MB/s | 1.1 MB 00:00 2020-03-11T22:35:47.961 INFO:teuthology.orchestra.run.smithi142.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 711 kB/s | 61 kB 00:00 2020-03-11T22:35:47.963 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:47.963 INFO:teuthology.orchestra.run.smithi142.stdout:Total 6.7 MB/s | 1.9 MB 00:00 2020-03-11T22:35:47.970 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:47.984 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:47.984 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:48.050 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:48.051 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:48.508 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:48.757 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:35:49.018 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:35:49.215 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:35:49.411 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:35:49.593 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:35:49.774 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:35:50.007 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:35:50.242 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:35:51.279 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:35:51.627 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:35:51.812 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:35:51.970 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:35:52.137 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:35:52.367 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:35:52.368 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:35:52.369 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:35:52.369 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:35:52.369 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:35:52.369 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:35:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:35:52.691 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:52.691 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:35:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:35:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:35:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:35:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:35:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:35:52.692 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:35:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:35:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:35:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:35:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:35:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:35:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:35:52.693 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:52.694 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:52.834 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:35:53.231 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:53.600 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:53.601 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:53.601 INFO:teuthology.orchestra.run.smithi142.stdout: Package ArchVersion Repository Size 2020-03-11T22:35:53.601 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:53.601 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:53.601 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:35:53.601 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout:Install 4 Packages 2020-03-11T22:35:53.602 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:53.603 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 461 k 2020-03-11T22:35:53.603 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 1.4 M 2020-03-11T22:35:53.603 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:53.664 INFO:teuthology.orchestra.run.smithi142.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 785 kB/s | 48 kB 00:00 2020-03-11T22:35:53.674 INFO:teuthology.orchestra.run.smithi142.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 4.9 MB/s | 356 kB 00:00 2020-03-11T22:35:53.721 INFO:teuthology.orchestra.run.smithi142.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 373 kB/s | 21 kB 00:00 2020-03-11T22:35:53.767 INFO:teuthology.orchestra.run.smithi142.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 221 kB/s | 36 kB 00:00 2020-03-11T22:35:53.768 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:53.769 INFO:teuthology.orchestra.run.smithi142.stdout:Total 2.7 MB/s | 461 kB 00:00 2020-03-11T22:35:53.772 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:53.779 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:53.779 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:53.808 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:53.809 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:53.958 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:54.187 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:35:54.474 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:35:54.659 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:35:54.804 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:35:55.000 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:35:55.000 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:35:55.000 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:35:55.000 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:35:55.222 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:35:55.222 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:55.222 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:55.222 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:35:55.223 INFO:teuthology.orchestra.run.smithi142.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:35:55.223 INFO:teuthology.orchestra.run.smithi142.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:35:55.223 INFO:teuthology.orchestra.run.smithi142.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:35:55.223 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:55.223 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:55.351 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-fuse 2020-03-11T22:35:55.749 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:56.132 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:56.133 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:56.133 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2020-03-11T22:35:56.133 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:56.133 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:56.133 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:35:56.134 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2020-03-11T22:35:56.134 INFO:teuthology.orchestra.run.smithi142.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:35:56.134 INFO:teuthology.orchestra.run.smithi142.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:35:56.134 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:56.134 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:56.134 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:56.135 INFO:teuthology.orchestra.run.smithi142.stdout:Install 3 Packages 2020-03-11T22:35:56.135 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:56.135 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 676 k 2020-03-11T22:35:56.136 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 2.1 M 2020-03-11T22:35:56.136 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:56.165 INFO:teuthology.orchestra.run.smithi142.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2020-03-11T22:35:56.173 INFO:teuthology.orchestra.run.smithi142.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 555 kB/s | 21 kB 00:00 2020-03-11T22:35:56.491 INFO:teuthology.orchestra.run.smithi142.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.6 MB/s | 572 kB 00:00 2020-03-11T22:35:56.492 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:56.493 INFO:teuthology.orchestra.run.smithi142.stdout:Total 1.8 MB/s | 676 kB 00:00 2020-03-11T22:35:56.497 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:56.517 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:56.518 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:56.582 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:56.583 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:56.858 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:56.858 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:35:56.858 INFO:teuthology.orchestra.run.smithi142.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:35:56.858 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:57.285 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:35:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:35:57.758 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:35:57.759 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:35:57.759 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:35:58.082 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:35:58.083 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:58.083 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:35:58.083 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:35:58.083 INFO:teuthology.orchestra.run.smithi142.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:35:58.083 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:58.083 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:35:58.369 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install librados-devel 2020-03-11T22:35:58.773 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:35:59.201 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:35:59.202 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:59.202 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2020-03-11T22:35:59.202 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:59.203 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:35:59.203 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:35:59.203 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:59.203 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:35:59.203 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:35:59.203 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2020-03-11T22:35:59.204 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:35:59.204 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 141 k 2020-03-11T22:35:59.204 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 487 k 2020-03-11T22:35:59.204 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:35:59.463 INFO:teuthology.orchestra.run.smithi142.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 541 kB/s | 141 kB 00:00 2020-03-11T22:35:59.464 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:35:59.464 INFO:teuthology.orchestra.run.smithi142.stdout:Total 538 kB/s | 141 kB 00:00 2020-03-11T22:35:59.466 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:35:59.473 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:35:59.473 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:35:59.521 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:35:59.521 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:35:59.628 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:35:59.818 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:00.026 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:00.305 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:00.306 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:00.306 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:36:00.306 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:36:00.306 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:00.306 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:00.586 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install libcephfs2 2020-03-11T22:36:00.996 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:01.285 INFO:teuthology.orchestra.run.smithi142.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:36:01.398 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:01.445 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:36:01.445 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:01.519 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install libcephfs-devel 2020-03-11T22:36:01.924 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:02.418 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:02.418 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2020-03-11T22:36:02.419 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:02.419 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:36:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:36:02.419 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:02.419 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:36:02.420 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:02.420 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2020-03-11T22:36:02.420 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:02.420 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 21 k 2020-03-11T22:36:02.420 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 76 k 2020-03-11T22:36:02.420 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:36:02.583 INFO:teuthology.orchestra.run.smithi142.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 130 kB/s | 21 kB 00:00 2020-03-11T22:36:02.584 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:36:02.584 INFO:teuthology.orchestra.run.smithi142.stdout:Total 129 kB/s | 21 kB 00:00 2020-03-11T22:36:02.586 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:36:02.590 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:36:02.590 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:36:02.592 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:36:02.593 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:36:02.646 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:36:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:02.965 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:03.173 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:03.173 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:03.173 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:36:03.173 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:36:03.173 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:03.174 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:03.316 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install librados2 2020-03-11T22:36:03.714 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:03.992 INFO:teuthology.orchestra.run.smithi142.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:36:04.099 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:04.100 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:36:04.100 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:04.217 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install librbd1 2020-03-11T22:36:04.613 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:04.883 INFO:teuthology.orchestra.run.smithi142.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:36:04.990 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:04.992 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:36:04.992 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:05.116 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rados 2020-03-11T22:36:05.521 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:05.800 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:36:05.905 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:05.906 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:36:05.906 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:06.033 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rgw 2020-03-11T22:36:06.430 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:06.700 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:36:06.805 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:06.806 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:36:06.806 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:06.940 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-cephfs 2020-03-11T22:36:07.344 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:07.608 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:36:07.714 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:07.715 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:36:07.715 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:07.833 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install python3-rbd 2020-03-11T22:36:08.230 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:08.491 INFO:teuthology.orchestra.run.smithi142.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:36:08.596 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:08.597 INFO:teuthology.orchestra.run.smithi142.stdout:Nothing to do. 2020-03-11T22:36:08.597 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:08.739 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-fuse 2020-03-11T22:36:09.137 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:09.513 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:09.514 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:09.514 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2020-03-11T22:36:09.515 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:09.515 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:36:09.515 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:36:09.515 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:09.515 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:36:09.515 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:09.515 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2020-03-11T22:36:09.516 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:09.516 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 72 k 2020-03-11T22:36:09.516 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 230 k 2020-03-11T22:36:09.516 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:36:09.738 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 324 kB/s | 72 kB 00:00 2020-03-11T22:36:09.739 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:36:09.739 INFO:teuthology.orchestra.run.smithi142.stdout:Total 322 kB/s | 72 kB 00:00 2020-03-11T22:36:09.741 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:36:09.752 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:36:09.752 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:36:09.799 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:36:09.800 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:36:09.944 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:36:10.099 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:10.298 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:10.505 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:10.505 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:10.506 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:36:10.506 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:36:10.506 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:10.506 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:10.783 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-mirror 2020-03-11T22:36:11.186 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:11.581 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:11.582 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:11.582 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repo Size 2020-03-11T22:36:11.582 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:11.582 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:36:11.582 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:36:11.583 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:11.583 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:36:11.583 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:11.583 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2020-03-11T22:36:11.583 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:11.583 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 2.7 M 2020-03-11T22:36:11.583 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 12 M 2020-03-11T22:36:11.584 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:36:11.968 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 7.0 MB/s | 2.7 MB 00:00 2020-03-11T22:36:11.968 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:36:11.969 INFO:teuthology.orchestra.run.smithi142.stdout:Total 6.9 MB/s | 2.7 MB 00:00 2020-03-11T22:36:11.971 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:36:11.983 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:36:11.984 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:36:12.041 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:36:12.041 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:36:12.400 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:36:13.168 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:13.501 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:13.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:13.746 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:13.746 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:36:13.746 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:36:13.746 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:13.746 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:14.044 INFO:teuthology.orchestra.run.smithi142:> sudo yum -y install rbd-nbd 2020-03-11T22:36:14.452 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:53 ago on Wed 11 Mar 2020 10:34:21 PM UTC. 2020-03-11T22:36:14.831 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2020-03-11T22:36:14.832 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:14.832 INFO:teuthology.orchestra.run.smithi142.stdout: Package Architecture Version Repository Size 2020-03-11T22:36:14.832 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:14.832 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2020-03-11T22:36:14.833 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:36:14.833 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:14.833 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2020-03-11T22:36:14.833 INFO:teuthology.orchestra.run.smithi142.stdout:================================================================================ 2020-03-11T22:36:14.833 INFO:teuthology.orchestra.run.smithi142.stdout:Install 1 Package 2020-03-11T22:36:14.833 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:14.833 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 141 k 2020-03-11T22:36:14.834 INFO:teuthology.orchestra.run.smithi142.stdout:Installed size: 526 k 2020-03-11T22:36:14.834 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2020-03-11T22:36:15.093 INFO:teuthology.orchestra.run.smithi142.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 543 kB/s | 141 kB 00:00 2020-03-11T22:36:15.093 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:36:15.093 INFO:teuthology.orchestra.run.smithi142.stdout:Total 541 kB/s | 141 kB 00:00 2020-03-11T22:36:15.095 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2020-03-11T22:36:15.107 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2020-03-11T22:36:15.107 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2020-03-11T22:36:15.156 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2020-03-11T22:36:15.156 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2020-03-11T22:36:15.280 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2020-03-11T22:36:15.473 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:15.687 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:15.936 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:36:15.937 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:15.937 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2020-03-11T22:36:15.937 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:36:15.937 INFO:teuthology.orchestra.run.smithi142.stdout: 2020-03-11T22:36:15.938 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2020-03-11T22:36:16.226 DEBUG:teuthology.parallel:result is None 2020-03-11T22:36:16.227 INFO:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2020-03-11T22:36:16.262 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS 2020-03-11T22:36:16.264 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:36:16.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:36:16.493 INFO:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:36:16.511 INFO:teuthology.orchestra.run.smithi142.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:36:16.512 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:36:16.512 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:36:16.512 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:36:16.520 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:36:16.581 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:36:16.582 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:36:16.650 INFO:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:36:16.716 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:36:16.717 INFO:teuthology.orchestra.run.smithi142:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:36:16.803 INFO:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:36:16.869 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:36:16.888 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:36:16.888 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:36:16.888 INFO:tasks.workunit:timeout=3h 2020-03-11T22:36:16.888 INFO:tasks.workunit:cleanup=True 2020-03-11T22:36:16.889 INFO:teuthology.orchestra.run.smithi142:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:36:16.922 INFO:teuthology.orchestra.run.smithi142.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:36:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:36:16.922 INFO:teuthology.orchestra.run.smithi142:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:36:16.976 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:36:16.976 INFO:teuthology.orchestra.run.smithi142:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:36:17.031 INFO:teuthology.orchestra.run.smithi142:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:36:17.137 INFO:tasks.workunit.client.0.smithi142.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:37:51.479 INFO:tasks.workunit.client.0.smithi142.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:37:51.479 INFO:tasks.workunit.client.0.smithi142.stderr: 2020-03-11T22:37:51.480 INFO:tasks.workunit.client.0.smithi142.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:37:51.480 INFO:tasks.workunit.client.0.smithi142.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:37:51.480 INFO:tasks.workunit.client.0.smithi142.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:37:51.480 INFO:tasks.workunit.client.0.smithi142.stderr: 2020-03-11T22:37:51.480 INFO:tasks.workunit.client.0.smithi142.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:37:51.480 INFO:tasks.workunit.client.0.smithi142.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:37:51.480 INFO:tasks.workunit.client.0.smithi142.stderr: 2020-03-11T22:37:51.481 INFO:tasks.workunit.client.0.smithi142.stderr: git checkout -b 2020-03-11T22:37:51.481 INFO:tasks.workunit.client.0.smithi142.stderr: 2020-03-11T22:37:51.481 INFO:tasks.workunit.client.0.smithi142.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:37:51.484 INFO:teuthology.orchestra.run.smithi142:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:37:51.555 DEBUG:teuthology.orchestra.remote:smithi142:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:37:51.603 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:37:51.604 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:37:51.604 INFO:teuthology.orchestra.run.smithi142:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:37:51.632 INFO:tasks.workunit.client.0.smithi142.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:37:51.642 INFO:tasks.workunit.client.0.smithi142.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:37:51.643 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:37:51.643 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:37:51.643 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:37:51.643 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:37:51.643 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:37:51.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:37:51.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:37:51.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:37:51.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:37:51.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:37:51.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:37:51.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:37:51.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:37:51.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:37:51.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:37:51.645 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:37:51.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:37:51.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:37:51.646 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:37:51.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:37:51.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:37:51.646 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:37:51.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:37:51.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:37:51.648 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:37:51.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:37:51.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:37:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:37:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:37:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:37:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:37:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:37:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:37:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:37:51.650 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:37:51.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none ' 2020-03-11T22:37:51.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:37:51.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:37:51.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:37:51.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:51.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:37:51.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:37:51.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:37:51.657 INFO:tasks.workunit.client.0.smithi142.stderr:TEST_divergent_2 2020-03-11T22:37:51.657 INFO:tasks.workunit.client.0.smithi142.stderr:TEST_divergent_3 2020-03-11T22:37:51.657 INFO:tasks.workunit.client.0.smithi142.stderr:TEST_divergent_ec' 2020-03-11T22:37:51.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:37:51.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:37:51.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:37:51.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:37:51.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:37:51.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:37:51.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:37:51.658 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:51.658 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:51.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:51.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:51.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:51.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:51.660 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:37:51.661 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:37:51.661 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:37:51.661 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:37:51.661 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:37:51.661 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:37:51.662 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:51.662 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:37:51.663 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:37:51.663 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:51.663 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:37:51.664 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:37:51.664 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:37:51.664 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:37:51.664 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:51.665 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:51.665 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:37:51.665 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:37:51.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:37:51.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:37:51.666 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:37:51.667 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:51.667 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:51.667 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17403 2020-03-11T22:37:51.668 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:37:51.668 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:37:51.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:37:51.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:37:51.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:37:51.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:37:51.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:37:51.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:37:51.670 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:37:51.670 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:37:51.670 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:37:51.670 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:37:51.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:37:51.675 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:37:51.676 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:37:51.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:37:51.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:37:51.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:37:51.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:37:51.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:37:51.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:37:51.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:37:51.983 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:37:51.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:51.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:51.984 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:51.984 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:51.984 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:51.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:37:51.985 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:37:52.135 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:37:52.137 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:37:52.137 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:37:52.137 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:37:52.137 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:37:52.138 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:37:52.138 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:37:52.138 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:37:52.138 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:37:52.139 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:52.139 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:52.139 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:52.139 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:37:52.140 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:37:52.140 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get fsid 2020-03-11T22:37:52.223 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:37:52.223 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:37:52.223 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:52.224 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:52.225 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:37:52.225 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:37:52.225 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get mon_host 2020-03-11T22:37:52.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:37:52.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:37:52.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:37:52.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:37:52.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:37:52.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:37:52.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:37:52.809 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:37:52.809 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:52.809 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:52.809 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:52.809 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:52.810 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:52.810 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:37:52.810 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:37:52.810 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:37:52.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:37:52.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:37:52.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:52.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:52.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:37:52.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:52.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:37:52.842 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:37:52.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:37:52.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:52.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:52.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:52.845 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:52.845 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:52.845 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:52.845 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:52.846 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:52.846 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:52.846 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:37:52.846 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:37:52.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:52.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:52.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:52.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:52.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:52.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:52.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:52.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:52.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:52.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:52.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:37:52.849 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:52.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3cdb6518-067a-4954-96ef-4e52b4a57c3d 2020-03-11T22:37:52.850 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 3cdb6518-067a-4954-96ef-4e52b4a57c3d' 2020-03-11T22:37:52.850 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:52.850 INFO:tasks.workunit.client.0.smithi142.stdout:add osd0 3cdb6518-067a-4954-96ef-4e52b4a57c3d 2020-03-11T22:37:52.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBAaGleTLrSMhAATVj6FTtpDqf4HrlDXw1sEw== 2020-03-11T22:37:52.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBAaGleTLrSMhAATVj6FTtpDqf4HrlDXw1sEw=="}' 2020-03-11T22:37:52.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3cdb6518-067a-4954-96ef-4e52b4a57c3d -i td/divergent-priors/0/new.json 2020-03-11T22:37:53.276 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:37:53.285 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:37:53.299 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBAaGleTLrSMhAATVj6FTtpDqf4HrlDXw1sEw== --osd-uuid 3cdb6518-067a-4954-96ef-4e52b4a57c3d 2020-03-11T22:37:53.330 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:53.327+0000 7f1542df9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:53.332 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:53.329+0000 7f1542df9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:53.334 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:53.331+0000 7f1542df9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:53.334 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:53.331+0000 7f1542df9ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:53.334 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:53.331+0000 7f1542df9ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:53.334 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:53.331+0000 7f1542df9ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:53.426 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:53.424+0000 7f1542df9ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:37:56.390 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:37:56.390 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:56.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:37:56.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:56.391 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd0 key to auth repository 2020-03-11T22:37:56.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:37:56.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:56.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:56.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:56.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:56.640 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:37:56.667 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:56.664+0000 7f148984cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:56.669 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:56.666+0000 7f148984cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:56.670 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:56.668+0000 7f148984cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:56.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:37:56.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:56.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:56.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:56.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:56.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:56.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:56.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:56.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:56.880 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:37:57.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:57.185 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:57.183+0000 7f148984cec0 -1 Falling back to public interface 2020-03-11T22:37:57.879 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:57.877+0000 7f148984cec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:37:58.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:58.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:58.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:58.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:58.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:58.125 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:37:58.538 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17784,v1:127.0.0.1:6803/17784] [v2:127.0.0.1:6804/17784,v1:127.0.0.1:6805/17784] exists,up 3cdb6518-067a-4954-96ef-4e52b4a57c3d 2020-03-11T22:37:58.538 INFO:tasks.workunit.client.0.smithi142.stdout:add osd1 2a110545-206c-4222-a307-a9facc224c3b 2020-03-11T22:37:58.538 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:58.538 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:58.538 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:58.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:37:58.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:37:58.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:58.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:58.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:37:58.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:58.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:58.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:58.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:58.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:58.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:58.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:58.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:58.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:58.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:58.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:58.541 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:58.541 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:58.541 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:58.541 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:58.542 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:58.542 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:37:58.542 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:37:58.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:37:58.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:58.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:58.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:58.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:58.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:58.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2a110545-206c-4222-a307-a9facc224c3b 2020-03-11T22:37:58.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 2a110545-206c-4222-a307-a9facc224c3b' 2020-03-11T22:37:58.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:58.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBGaGleMq7wIBAA1MQCglJyqEBjwllVqxclxQ== 2020-03-11T22:37:58.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBGaGleMq7wIBAA1MQCglJyqEBjwllVqxclxQ=="}' 2020-03-11T22:37:58.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2a110545-206c-4222-a307-a9facc224c3b -i td/divergent-priors/1/new.json 2020-03-11T22:37:59.074 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:37:59.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:37:59.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBGaGleMq7wIBAA1MQCglJyqEBjwllVqxclxQ== --osd-uuid 2a110545-206c-4222-a307-a9facc224c3b 2020-03-11T22:37:59.115 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:59.107+0000 7f284427bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:59.124 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:59.117+0000 7f284427bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:59.126 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:59.119+0000 7f284427bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:59.126 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:59.119+0000 7f284427bec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:59.127 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:59.119+0000 7f284427bec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:59.127 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:59.119+0000 7f284427bec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:59.187 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:37:59.175+0000 7f284427bec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:38:02.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:38:02.140 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:38:02.140 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd1 key to auth repository 2020-03-11T22:38:02.141 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:38:02.141 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:38:02.595 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:38:02.595 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:38:02.595 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:02.596 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:38:02.596 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:38:02.596 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:38:02.621 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:02.618+0000 7f6bf7f25ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:02.623 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:02.620+0000 7f6bf7f25ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:02.625 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:02.622+0000 7f6bf7f25ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:02.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:38:02.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:02.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:38:02.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:02.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:02.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:02.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:02.942 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:02.942 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:02.942 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:38:03.141 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:03.138+0000 7f6bf7f25ec0 -1 Falling back to public interface 2020-03-11T22:38:03.291 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:03.819 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:03.816+0000 7f6bf7f25ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:38:04.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:04.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:04.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:04.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:04.293 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:38:04.293 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:38:04.652 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18195,v1:127.0.0.1:6811/18195] [v2:127.0.0.1:6812/18195,v1:127.0.0.1:6813/18195] exists,up 2a110545-206c-4222-a307-a9facc224c3b 2020-03-11T22:38:04.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:04.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:04.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:04.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:38:04.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:38:04.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:38:04.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:38:04.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:38:04.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:38:04.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:38:04.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:04.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:04.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:04.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:04.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:38:04.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:38:04.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:04.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:38:04.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:04.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:38:04.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:04.656 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:04.656 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:04.656 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:04.656 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:38:04.658 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:38:04.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:38:04.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:04.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:38:04.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:38:04.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:04.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:04.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:04.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:04.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:04.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:38:04.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:38:04.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:38:04.660 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:38:04.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2971b697-7a4b-4c40-bd48-ff00e2071e11 2020-03-11T22:38:04.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 2971b697-7a4b-4c40-bd48-ff00e2071e11' 2020-03-11T22:38:04.660 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:38:04.661 INFO:tasks.workunit.client.0.smithi142.stdout:add osd2 2971b697-7a4b-4c40-bd48-ff00e2071e11 2020-03-11T22:38:04.673 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBMaGleavj6JxAAqu6Xy2abmeWpySmhdLweCQ== 2020-03-11T22:38:04.673 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBMaGleavj6JxAAqu6Xy2abmeWpySmhdLweCQ=="}' 2020-03-11T22:38:04.673 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2971b697-7a4b-4c40-bd48-ff00e2071e11 -i td/divergent-priors/2/new.json 2020-03-11T22:38:05.181 INFO:tasks.workunit.client.0.smithi142.stdout:2 2020-03-11T22:38:05.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:38:05.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBMaGleavj6JxAAqu6Xy2abmeWpySmhdLweCQ== --osd-uuid 2971b697-7a4b-4c40-bd48-ff00e2071e11 2020-03-11T22:38:05.220 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:05.217+0000 7fa3aba34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:05.234 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:05.231+0000 7fa3aba34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:05.235 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:05.232+0000 7fa3aba34ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:05.233+0000 7fa3aba34ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:05.233+0000 7fa3aba34ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:38:05.236 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:05.233+0000 7fa3aba34ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:38:05.301 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:05.298+0000 7fa3aba34ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:38:08.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:38:08.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:38:08.306 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd2 key to auth repository 2020-03-11T22:38:08.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:38:08.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:38:08.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:38:08.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:38:08.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:38:08.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:38:08.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:08.762 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:38:08.788 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:08.784+0000 7fb41bfe0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:08.789 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:08.786+0000 7fb41bfe0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:08.790 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:08.787+0000 7fb41bfe0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:09.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:38:09.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:09.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:09.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:09.110 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:09.110 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:09.110 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:09.110 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:09.110 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:09.110 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:38:09.307 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:09.304+0000 7fb41bfe0ec0 -1 Falling back to public interface 2020-03-11T22:38:09.458 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:09.999 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:09.995+0000 7fb41bfe0ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:38:10.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:10.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:10.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:10.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:10.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:10.461 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:38:10.808 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/18558,v1:127.0.0.1:6819/18558] [v2:127.0.0.1:6820/18558,v1:127.0.0.1:6821/18558] exists,up 2971b697-7a4b-4c40-bd48-ff00e2071e11 2020-03-11T22:38:10.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:10.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:10.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:10.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:38:11.551 INFO:tasks.workunit.client.0.smithi142.stderr:noout is set 2020-03-11T22:38:11.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:38:12.169 INFO:tasks.workunit.client.0.smithi142.stderr:noin is set 2020-03-11T22:38:12.177 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:38:12.988 INFO:tasks.workunit.client.0.smithi142.stderr:nodown is set 2020-03-11T22:38:12.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:38:12.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:38:13.494 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' already exists 2020-03-11T22:38:13.501 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:38:14.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:38:15.074 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 size to 3 2020-03-11T22:38:15.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:38:15.734 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 min_size to 2 2020-03-11T22:38:15.742 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:38:15.742 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:15.742 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:16.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:16.100 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:38:16.100 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:38:16.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:16.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:16.101 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:16.230 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:38:16.230 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:38:16.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:16.231 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:16.361 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:38:16.361 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2020-03-11T22:38:16.361 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:16.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:16.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:38:16.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2020-03-11T22:38:16.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:16.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:38:16.491 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:16.491 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:16.492 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:38:16.492 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:16.493 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:38:16.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:38:16.494 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:38:16.494 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:16.840 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-11T22:38:16.840 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:17.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:17.842 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:18.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:38:18.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:18.198 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:38:18.198 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:18.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:18.200 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:38:18.200 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:18.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:38:18.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:38:18.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:18.202 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:38:18.545 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:38:18.545 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:18.546 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:38:18.546 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:18.546 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:18.548 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:38:18.548 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:18.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:38:18.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:38:18.550 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:18.550 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:38:18.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:38:18.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:38:18.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:18.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:18.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:18.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:18.899 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:38:18.899 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:18.899 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:18.899 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:18.900 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:18.900 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:18.988 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:18.988 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:18.988 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:18.988 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:18.989 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:19.332 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:19.332 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:38:19.332 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:38:19.332 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:19.332 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:19.333 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:19.462 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:38:19.462 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:38:19.462 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:19.462 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:19.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672967 2020-03-11T22:38:19.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672967' 2020-03-11T22:38:19.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:19.592 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:19.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:38:19.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672967 2-64424509446' 2020-03-11T22:38:19.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:19.721 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:38:19.721 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:19.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:19.723 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:38:19.723 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:19.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:38:19.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:38:19.724 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:38:19.725 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:20.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836490 2020-03-11T22:38:20.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:21.070 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:21.071 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:21.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:38:21.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:21.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672967 2020-03-11T22:38:21.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:21.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:21.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672967 2020-03-11T22:38:21.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:21.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672967 2020-03-11T22:38:21.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2020-03-11T22:38:21.422 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672967 2020-03-11T22:38:21.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:21.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672967 2020-03-11T22:38:21.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:21.788 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:38:21.788 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:21.789 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:21.790 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:38:21.790 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:21.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:38:21.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:38:21.792 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:22.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-11T22:38:22.136 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:22.137 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:22.137 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:22.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:22.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:22.585 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:22.585 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:22.585 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:22.586 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:22.586 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:22.586 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:22.918 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:22.919 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:22.919 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:22.920 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:23.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:23.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:23.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:23.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:38:23.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:38:23.685 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:38:23.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:38:23.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:38:23.695 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:23.695 INFO:tasks.workunit.client.0.smithi142.stdout:primary and soon to be divergent is 1 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:24.016 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:38:14.011020+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:38:24.017 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:24.018 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:38:24.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:38:24.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi142.stdout:writing initial objects 2020-03-11T22:38:24.029 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.071 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:38:24.156 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.157 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:38:24.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:38:24.239 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.488 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:38:24.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:38:24.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:38:24.492 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.493 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:38:24.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:38:24.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:38:24.821 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:24.821 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:38:25.296 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:25.297 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:38:25.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:25.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:38:25.638 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:25.638 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:38:25.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:25.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:38:26.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:26.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:38:26.222 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:26.222 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:38:26.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:26.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:38:26.556 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:26.556 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:38:26.902 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:26.902 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:38:26.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:26.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:38:27.008 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.008 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:38:27.077 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.078 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:38:27.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:38:27.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:38:27.328 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.328 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:38:27.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:38:27.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:38:27.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:38:27.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:38:27.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:38:27.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:38:27.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:38:27.737 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.737 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:38:27.777 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.777 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:38:27.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:38:27.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:38:27.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:27.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:38:28.037 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.037 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:38:28.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:38:28.135 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.135 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:38:28.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:38:28.254 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.255 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:38:28.297 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.297 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:38:28.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:38:28.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:38:28.455 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.455 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:38:28.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:38:28.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:38:28.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:38:28.663 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.663 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:38:28.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:38:28.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:38:28.788 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.788 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:38:28.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:38:28.913 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:28.913 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:38:29.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:38:29.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:38:29.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:38:29.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:38:29.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:38:29.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:38:29.361 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.361 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:38:29.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:38:29.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:38:29.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:38:29.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:38:29.595 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.595 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:38:29.674 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.674 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:38:29.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:38:29.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:38:29.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:38:29.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:38:29.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:38:29.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:38:29.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:29.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:38:30.036 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.037 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:38:30.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:38:30.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.156 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:38:30.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:38:30.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:38:30.342 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.342 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:38:30.381 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.381 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:38:30.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:38:30.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:38:30.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:38:30.581 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.581 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:38:30.620 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.620 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:38:30.702 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.703 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:38:30.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:38:30.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.798 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:38:30.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:38:30.917 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:30.917 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:38:31.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:31.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:38:31.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:31.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:38:31.245 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:31.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:38:31.297 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:31.298 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:38:31.338 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:31.339 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:38:31.381 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:38:31.381 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:38:31.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:38:31.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:38:31.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:31.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:31.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:31.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:31.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:38:31.422 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:31.423 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:31.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:31.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:31.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:31.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:31.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:31.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:31.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:31.478 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:31.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:31.819 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:38:31.819 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:38:31.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:31.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:31.819 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:31.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:38:31.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:38:31.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:31.949 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:32.079 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:38:32.079 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2020-03-11T22:38:32.079 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:32.080 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:32.208 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509451 2020-03-11T22:38:32.208 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509451' 2020-03-11T22:38:32.209 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:32.209 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:38:32.209 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:32.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:32.211 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:38:32.211 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:32.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:38:32.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:38:32.212 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:38:32.213 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:32.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836494 2020-03-11T22:38:32.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:33.556 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:33.557 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:33.904 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-11T22:38:33.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:33.905 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:38:33.905 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:33.906 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:33.907 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:38:33.907 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:33.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:38:33.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:38:33.908 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:38:33.909 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:34.277 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:38:34.277 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:34.277 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509451 2020-03-11T22:38:34.278 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:34.278 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:34.279 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509451 2020-03-11T22:38:34.279 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:34.280 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509451 2020-03-11T22:38:34.280 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2020-03-11T22:38:34.281 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509451 2020-03-11T22:38:34.281 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:34.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509451 2020-03-11T22:38:34.656 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:34.657 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:34.657 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:35.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:35.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:35.107 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:35.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:35.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:35.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:35.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:35.109 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:35.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:35.439 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:35.440 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:35.440 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:35.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:35.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:35.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:35.881 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:38:35.881 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:38:35.881 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:38:35.881 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:38:35.882 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:38:36.225 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:38:36.225 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:38:36.225 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:36.226 INFO:tasks.workunit.client.0.smithi142.stdout:blackholing osds 0 2 2020-03-11T22:38:36.545 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:36.545 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:38:14.011020+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:38:36.545 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:38:36.545 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:36.546 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:38:36.554 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:36.555 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:38:36.555 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:36.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:36.556 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:36.556 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:36.556 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:38:36.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:38:36.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:38:36.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:38:36.642 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:38:36.642 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:36.642 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:38:36.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:36.650 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:38:36.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:36.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:36.651 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:36.651 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:36.651 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:38:36.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:38:36.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:38:36.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:38:36.737 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:38:36.737 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:36.737 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:38:36.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:38:36.745 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:38:36.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:38:36.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:38:36.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:38:36.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:38:36.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:38:36.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:38:36.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:38:36.747 INFO:tasks.workunit.client.0.smithi142.stdout:writing divergent object 2020-03-11T22:38:36.748 INFO:tasks.workunit.client.0.smithi142.stdout:create missing divergent object 2020-03-11T22:38:36.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:38:36.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:38:36.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:38:36.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:38:36.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:38:36.751 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:38:36.751 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:38:36.751 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:38:36.751 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:38:36.751 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:38:36.752 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:38:36.752 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:38:36.752 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:38:36.752 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:38:36.752 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:38:36.752 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:38:37.105 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:38:37.106 INFO:tasks.workunit.client.0.smithi142.stderr:0 2020-03-11T22:38:37.106 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:38:37.106 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:38:37.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:38:37.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:38:37.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:38:37.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:38:37.107 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:38:37.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:38:37.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:38:37.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:38:37.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:38:37.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:38:37.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:38:37.109 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:38:37.110 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:38:37.110 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:37.110 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:37.110 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:37.110 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:37.110 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:38:37.110 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:38:37.111 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17403/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:38:37.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:38:37.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:38:37.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:38:37.205 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:38:37.205 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:38:37.205 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:38:37.206 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:37.206 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:37.206 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:38:37.206 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:38:37.206 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:38:37.207 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:38:37.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:38:37.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:38:47.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:38:47.321 INFO:tasks.workunit.client.0.smithi142.stderr:rados: no process found 2020-03-11T22:38:47.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:38:47.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:47.322 INFO:tasks.workunit.client.0.smithi142.stdout:killing all the osds 2020-03-11T22:38:47.652 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:47.653 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:38:37.545987+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:38:47.653 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:38:47.653 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:47.654 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:38:47.662 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:38:47.663 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:47.663 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:47.663 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:47.663 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:47.664 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:47.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:47.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:38:47.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:38:48.260 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already down. 2020-03-11T22:38:48.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:38:48.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:38:48.761 INFO:tasks.workunit.client.0.smithi142.stderr:osd.1 is already down. 2020-03-11T22:38:48.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:38:48.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:38:49.340 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already down. 2020-03-11T22:38:49.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:49.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:38:49.848 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already out. 2020-03-11T22:38:49.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:49.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:38:50.346 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already out. 2020-03-11T22:38:50.354 INFO:tasks.workunit.client.0.smithi142.stdout:bringing up non_divergent 0 2 2020-03-11T22:38:50.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:38:50.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:38:50.673 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:50.673 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:38:37.545987+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:38:50.674 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:38:50.674 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:50.675 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:38:50.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:50.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:38:50.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:50.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:50.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:38:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:38:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:38:50.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:38:50.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:50.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:50.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:50.689 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:50.689 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:50.689 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:50.690 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:50.690 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:50.690 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:38:50.690 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:38:50.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:38:50.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:50.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:50.691 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:50.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:50.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:50.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:50.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:50.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:50.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:38:50.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:38:50.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:50.694 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:38:50.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:38:50.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:50.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:50.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:50.694 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:38:50.712 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:50.709+0000 7fbed7f5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:50.714 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:50.711+0000 7fbed7f5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:50.718 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:50.713+0000 7fbed7f5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:51.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:38:51.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:51.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:51.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:51.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:51.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:51.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:51.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:51.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:51.040 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:38:51.231 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:51.228+0000 7fbed7f5fec0 -1 Falling back to public interface 2020-03-11T22:38:51.382 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:52.065 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:52.063+0000 7fbed7f5fec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:38:52.383 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:52.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:52.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:52.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:52.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:52.385 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:38:52.734 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22580,v1:127.0.0.1:6803/22580] [v2:127.0.0.1:6804/22580,v1:127.0.0.1:6805/22580] exists,up 3cdb6518-067a-4954-96ef-4e52b4a57c3d 2020-03-11T22:38:52.734 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:38:52.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:52.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:52.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:52.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:52.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:38:52.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:52.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:52.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:38:52.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:52.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:38:52.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:52.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:52.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:52.737 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:52.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:38:52.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:38:52.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:52.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:52.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:52.741 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:52.742 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:52.742 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:52.742 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:52.742 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:52.742 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:38:52.742 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:38:52.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:38:52.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:52.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:52.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:52.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:52.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:52.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:52.744 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:52.744 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:52.744 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:38:52.744 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:38:52.744 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:52.745 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:38:52.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:38:52.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:52.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:52.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:52.758 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:52.756+0000 7fa6bd71bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:52.760 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:52.758+0000 7fa6bd71bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:52.762 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:52.759+0000 7fa6bd71bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:53.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:38:53.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:53.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:53.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:53.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:53.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:53.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:53.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:53.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:53.105 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:38:53.278 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:53.276+0000 7fa6bd71bec0 -1 Falling back to public interface 2020-03-11T22:38:53.450 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:54.088 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:38:54.086+0000 7fa6bd71bec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:38:54.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:54.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:54.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:54.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:54.524 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:38:54.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:54.803 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22774,v1:127.0.0.1:6811/22774] [v2:127.0.0.1:6812/22774,v1:127.0.0.1:6813/22774] exists,up 2971b697-7a4b-4c40-bd48-ff00e2071e11 2020-03-11T22:38:54.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:54.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:54.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:54.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:54.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:38:55.346 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already in. 2020-03-11T22:38:55.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:38:55.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:38:55.850 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already in. 2020-03-11T22:38:55.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:38:55.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:38:55.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:55.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:55.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:55.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:55.859 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:38:55.860 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:55.860 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:55.860 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:55.860 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:55.860 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:55.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:55.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:55.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:55.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:55.915 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:56.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:56.271 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:38:56.271 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:38:56.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:56.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:56.271 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:56.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:38:56.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:38:56.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:56.401 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:56.491 INFO:tasks.workunit.client.0.smithi142.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:38:56.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:38:56.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:38:56.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:56.499 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:56.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:38:56.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:38:56.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:56.628 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:38:56.628 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:56.628 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:56.632 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:38:56.632 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:56.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:38:56.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:38:56.632 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:56.633 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:38:57.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:38:57.005 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:57.997 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:57.998 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:58.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888067 -lt 146028888067 2020-03-11T22:38:58.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:58.350 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:38:58.350 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:58.350 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:58.351 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:38:58.351 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:58.352 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:38:58.352 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:38:58.352 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:58.353 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 2020-03-11T22:38:58.696 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:38:58.696 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:38:58.696 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:58.697 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:38:58.697 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:58.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:58.698 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:38:58.698 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:58.699 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:38:58.699 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:38:58.699 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:38:58.699 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:59.044 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:38:59.044 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:59.045 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:59.045 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:59.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:59.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:59.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:59.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:59.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:59.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:59.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:59.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:59.817 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:59.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:59.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:59.818 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:00.259 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:39:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:39:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:39:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:39:00.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:00.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:00.361 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:00.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:00.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:00.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:00.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:00.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:00.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:00.690 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:00.691 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:00.691 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:01.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:01.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:01.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:01.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:01.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:01.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:01.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:01.134 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:01.134 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:01.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:01.573 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:01.573 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:39:01.573 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:01.573 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:39:01.774 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:01.774 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:01.774 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:01.774 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:01.774 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:01.775 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:01.775 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:01.775 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:02.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:02.105 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:02.106 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:02.106 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:02.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:02.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:02.548 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:02.548 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:02.548 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:02.548 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:02.548 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:02.548 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:02.549 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:02.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:02.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:02.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:39:02.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:02.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:39:03.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:03.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:03.393 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:03.393 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:03.394 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:03.394 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:03.394 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:03.394 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:03.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:03.725 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:03.725 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:03.725 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:04.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:04.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:04.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:04.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:04.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:04.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:04.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:04.170 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:04.171 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:04.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:04.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:04.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:39:04.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:04.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:39:05.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:05.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:05.412 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:05.413 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:05.413 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:05.413 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:05.413 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:05.413 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:05.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:05.753 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:05.753 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:05.754 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:06.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:06.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:06.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:06.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:06.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:06.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:06.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:06.194 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:06.194 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:06.634 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:06.634 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:06.634 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:39:06.635 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:06.635 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:39:08.235 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:08.235 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:08.236 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:08.236 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:08.236 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:08.237 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:08.237 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:08.237 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:08.577 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:08.578 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:08.578 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:08.579 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:09.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:09.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:09.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:09.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:09.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:09.021 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:09.021 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:09.021 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:09.021 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:39:09.472 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:09.473 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:39:12.673 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:12.673 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:12.674 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:12.674 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:12.674 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:12.674 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:12.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:12.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:13.019 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:13.019 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:13.020 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:13.020 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:13.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:13.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:13.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:13.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:13.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:13.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:13.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:13.464 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:13.464 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:13.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:13.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:13.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:39:13.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:13.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:39:20.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:20.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:20.311 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:20.311 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:20.311 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:20.311 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:20.312 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:20.312 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:20.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:20.649 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:20.650 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:20.650 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:21.093 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:21.093 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:21.093 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:21.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:21.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:21.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:21.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:21.094 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:21.094 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:21.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:21.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:21.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:39:21.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:21.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:39:28.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:28.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:28.844 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:28.844 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:28.844 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:28.844 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:28.844 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:28.845 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:29.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:29.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:29.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:29.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:29.631 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:29.631 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:29.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:29.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:29.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:29.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:29.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:29.632 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:29.633 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:30.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:30.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:30.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:39:30.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:39:30.515 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:39:30.515 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_fingerprint": "08513974-a0f8-448f-8f5f-a6466052783a", 2020-03-11T22:39:30.516 INFO:tasks.workunit.client.0.smithi142.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:39:30.516 INFO:tasks.workunit.client.0.smithi142.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:39:30.516 INFO:tasks.workunit.client.0.smithi142.stdout: "timestamp": "2020-03-11T22:39:30.511998+0000", 2020-03-11T22:39:30.516 INFO:tasks.workunit.client.0.smithi142.stdout: "tag": "", 2020-03-11T22:39:30.517 INFO:tasks.workunit.client.0.smithi142.stdout: "health": { 2020-03-11T22:39:30.517 INFO:tasks.workunit.client.0.smithi142.stdout: "status": "HEALTH_WARN", 2020-03-11T22:39:30.517 INFO:tasks.workunit.client.0.smithi142.stdout: "checks": { 2020-03-11T22:39:30.517 INFO:tasks.workunit.client.0.smithi142.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:39:30.518 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:30.518 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:39:30.518 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:39:30.518 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 17 2020-03-11T22:39:30.518 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.519 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [], 2020-03-11T22:39:30.519 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:39:30.519 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.519 INFO:tasks.workunit.client.0.smithi142.stdout: "OSD_DOWN": { 2020-03-11T22:39:30.520 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:30.520 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:39:30.520 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 osds down", 2020-03-11T22:39:30.520 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:39:30.521 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.521 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:39:30.521 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.521 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "osd.1 (root=default,host=smithi142) is down" 2020-03-11T22:39:30.522 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.522 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.522 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:39:30.522 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.523 INFO:tasks.workunit.client.0.smithi142.stdout: "PG_DEGRADED": { 2020-03-11T22:39:30.523 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:30.523 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:39:30.523 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:39:30.524 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:39:30.524 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.524 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:39:30.524 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.525 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:30.525 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.525 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.525 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:39:30.526 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.526 INFO:tasks.workunit.client.0.smithi142.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:39:30.526 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:30.527 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:39:30.527 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:39:30.527 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:39:30.528 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.528 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:39:30.528 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.529 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:39:30.529 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.529 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.529 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:39:30.530 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.530 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.530 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:39:30.530 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.531 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.531 INFO:tasks.workunit.client.0.smithi142.stdout: "mutes": [] 2020-03-11T22:39:30.531 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.531 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_first_committed": 1, 2020-03-11T22:39:30.532 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_last_committed": 1, 2020-03-11T22:39:30.532 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap": { 2020-03-11T22:39:30.532 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:39:30.532 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:39:30.533 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:37:51.727037Z", 2020-03-11T22:39:30.533 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:37:51.727037Z", 2020-03-11T22:39:30.533 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release": 15, 2020-03-11T22:39:30.534 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:39:30.534 INFO:tasks.workunit.client.0.smithi142.stdout: "features": { 2020-03-11T22:39:30.534 INFO:tasks.workunit.client.0.smithi142.stdout: "persistent": [ 2020-03-11T22:39:30.535 INFO:tasks.workunit.client.0.smithi142.stdout: "kraken", 2020-03-11T22:39:30.535 INFO:tasks.workunit.client.0.smithi142.stdout: "luminous", 2020-03-11T22:39:30.535 INFO:tasks.workunit.client.0.smithi142.stdout: "mimic", 2020-03-11T22:39:30.536 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap-prune", 2020-03-11T22:39:30.536 INFO:tasks.workunit.client.0.smithi142.stdout: "nautilus", 2020-03-11T22:39:30.537 INFO:tasks.workunit.client.0.smithi142.stdout: "octopus" 2020-03-11T22:39:30.537 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.538 INFO:tasks.workunit.client.0.smithi142.stdout: "optional": [] 2020-03-11T22:39:30.538 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.538 INFO:tasks.workunit.client.0.smithi142.stdout: "mons": [ 2020-03-11T22:39:30.538 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.539 INFO:tasks.workunit.client.0.smithi142.stdout: "rank": 0, 2020-03-11T22:39:30.539 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "a", 2020-03-11T22:39:30.539 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:39:30.539 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.540 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.540 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.542 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:39:30.542 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 0 2020-03-11T22:39:30.543 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.543 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.543 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.543 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:39:30.544 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:39:30.544 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 0, 2020-03-11T22:39:30.544 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 0 2020-03-11T22:39:30.545 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.545 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.545 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.545 INFO:tasks.workunit.client.0.smithi142.stdout: "quorum": [ 2020-03-11T22:39:30.546 INFO:tasks.workunit.client.0.smithi142.stdout: 0 2020-03-11T22:39:30.546 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.546 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap": { 2020-03-11T22:39:30.547 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 38, 2020-03-11T22:39:30.547 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:39:30.547 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:37:52.371668+0000", 2020-03-11T22:39:30.547 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:38:55.891563+0000", 2020-03-11T22:39:30.548 INFO:tasks.workunit.client.0.smithi142.stdout: "last_up_change": "2020-03-11T22:38:54.163813+0000", 2020-03-11T22:39:30.548 INFO:tasks.workunit.client.0.smithi142.stdout: "last_in_change": "2020-03-11T22:38:55.632536+0000", 2020-03-11T22:39:30.548 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:39:30.548 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_num": 5800384, 2020-03-11T22:39:30.549 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_set": [ 2020-03-11T22:39:30.549 INFO:tasks.workunit.client.0.smithi142.stdout: "nodown", 2020-03-11T22:39:30.549 INFO:tasks.workunit.client.0.smithi142.stdout: "noin", 2020-03-11T22:39:30.550 INFO:tasks.workunit.client.0.smithi142.stdout: "noout", 2020-03-11T22:39:30.550 INFO:tasks.workunit.client.0.smithi142.stdout: "pglog_hardlimit", 2020-03-11T22:39:30.550 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snapdirs", 2020-03-11T22:39:30.551 INFO:tasks.workunit.client.0.smithi142.stdout: "recovery_deletes", 2020-03-11T22:39:30.551 INFO:tasks.workunit.client.0.smithi142.stdout: "sortbitwise" 2020-03-11T22:39:30.551 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.551 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_version": 7, 2020-03-11T22:39:30.552 INFO:tasks.workunit.client.0.smithi142.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:39:30.552 INFO:tasks.workunit.client.0.smithi142.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:39:30.552 INFO:tasks.workunit.client.0.smithi142.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:39:30.552 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_snapshot": "", 2020-03-11T22:39:30.553 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_max": 1, 2020-03-11T22:39:30.553 INFO:tasks.workunit.client.0.smithi142.stdout: "max_osd": 3, 2020-03-11T22:39:30.553 INFO:tasks.workunit.client.0.smithi142.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:39:30.553 INFO:tasks.workunit.client.0.smithi142.stdout: "min_compat_client": "jewel", 2020-03-11T22:39:30.554 INFO:tasks.workunit.client.0.smithi142.stdout: "require_osd_release": "octopus", 2020-03-11T22:39:30.554 INFO:tasks.workunit.client.0.smithi142.stdout: "pools": [ 2020-03-11T22:39:30.554 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.554 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:39:30.555 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_name": "test", 2020-03-11T22:39:30.555 INFO:tasks.workunit.client.0.smithi142.stdout: "create_time": "2020-03-11T22:38:13.290482+0000", 2020-03-11T22:39:30.555 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": 1, 2020-03-11T22:39:30.555 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_names": "hashpspool", 2020-03-11T22:39:30.556 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:39:30.556 INFO:tasks.workunit.client.0.smithi142.stdout: "size": 3, 2020-03-11T22:39:30.556 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 2, 2020-03-11T22:39:30.556 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_rule": 0, 2020-03-11T22:39:30.557 INFO:tasks.workunit.client.0.smithi142.stdout: "object_hash": 2, 2020-03-11T22:39:30.557 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:39:30.557 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num": 1, 2020-03-11T22:39:30.557 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num": 1, 2020-03-11T22:39:30.558 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num_target": 1, 2020-03-11T22:39:30.558 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_target": 1, 2020-03-11T22:39:30.558 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_pending": 1, 2020-03-11T22:39:30.558 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pg_merge_meta": { 2020-03-11T22:39:30.559 INFO:tasks.workunit.client.0.smithi142.stdout: "source_pgid": "0.0", 2020-03-11T22:39:30.559 INFO:tasks.workunit.client.0.smithi142.stdout: "ready_epoch": 0, 2020-03-11T22:39:30.559 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_started": 0, 2020-03-11T22:39:30.559 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_clean": 0, 2020-03-11T22:39:30.560 INFO:tasks.workunit.client.0.smithi142.stdout: "source_version": "0'0", 2020-03-11T22:39:30.560 INFO:tasks.workunit.client.0.smithi142.stdout: "target_version": "0'0" 2020-03-11T22:39:30.560 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.560 INFO:tasks.workunit.client.0.smithi142.stdout: "last_change": "28", 2020-03-11T22:39:30.561 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend": "0", 2020-03-11T22:39:30.561 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:39:30.561 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:39:30.561 INFO:tasks.workunit.client.0.smithi142.stdout: "auid": 0, 2020-03-11T22:39:30.562 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:39:30.562 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_seq": 0, 2020-03-11T22:39:30.562 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_epoch": 0, 2020-03-11T22:39:30.563 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_snaps": [], 2020-03-11T22:39:30.563 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps": "[]", 2020-03-11T22:39:30.563 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_bytes": 0, 2020-03-11T22:39:30.563 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_objects": 0, 2020-03-11T22:39:30.564 INFO:tasks.workunit.client.0.smithi142.stdout: "tiers": [], 2020-03-11T22:39:30.564 INFO:tasks.workunit.client.0.smithi142.stdout: "tier_of": -1, 2020-03-11T22:39:30.564 INFO:tasks.workunit.client.0.smithi142.stdout: "read_tier": -1, 2020-03-11T22:39:30.565 INFO:tasks.workunit.client.0.smithi142.stdout: "write_tier": -1, 2020-03-11T22:39:30.565 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_mode": "none", 2020-03-11T22:39:30.565 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_bytes": 0, 2020-03-11T22:39:30.565 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_objects": 0, 2020-03-11T22:39:30.566 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:39:30.566 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:39:30.566 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:39:30.566 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_flush_age": 0, 2020-03-11T22:39:30.567 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_evict_age": 0, 2020-03-11T22:39:30.567 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profile": "", 2020-03-11T22:39:30.567 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_params": { 2020-03-11T22:39:30.567 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "none" 2020-03-11T22:39:30.567 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.568 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_period": 0, 2020-03-11T22:39:30.568 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_count": 0, 2020-03-11T22:39:30.568 INFO:tasks.workunit.client.0.smithi142.stdout: "use_gmt_hitset": true, 2020-03-11T22:39:30.568 INFO:tasks.workunit.client.0.smithi142.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:39:30.569 INFO:tasks.workunit.client.0.smithi142.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:39:30.569 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:39:30.569 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:39:30.570 INFO:tasks.workunit.client.0.smithi142.stdout: "grade_table": [], 2020-03-11T22:39:30.570 INFO:tasks.workunit.client.0.smithi142.stdout: "stripe_width": 0, 2020-03-11T22:39:30.570 INFO:tasks.workunit.client.0.smithi142.stdout: "expected_num_objects": 0, 2020-03-11T22:39:30.570 INFO:tasks.workunit.client.0.smithi142.stdout: "fast_read": false, 2020-03-11T22:39:30.571 INFO:tasks.workunit.client.0.smithi142.stdout: "options": {}, 2020-03-11T22:39:30.571 INFO:tasks.workunit.client.0.smithi142.stdout: "application_metadata": {} 2020-03-11T22:39:30.571 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.571 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.572 INFO:tasks.workunit.client.0.smithi142.stdout: "osds": [ 2020-03-11T22:39:30.572 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.572 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:39:30.573 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "3cdb6518-067a-4954-96ef-4e52b4a57c3d", 2020-03-11T22:39:30.573 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:39:30.573 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:39:30.573 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:39:30.574 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:39:30.574 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 5, 2020-03-11T22:39:30.574 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:39:30.574 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 34, 2020-03-11T22:39:30.575 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 37, 2020-03-11T22:39:30.575 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 29, 2020-03-11T22:39:30.575 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:39:30.575 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:39:30.576 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.576 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.576 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.576 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:39:30.577 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.577 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.577 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.577 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.578 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:39:30.578 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.578 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.578 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.578 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.578 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:39:30.578 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.579 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.579 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.579 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:39:30.579 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.579 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.579 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.579 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.580 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:39:30.580 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.580 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.580 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.580 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.580 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:30.580 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.581 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.581 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.581 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:39:30.581 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.581 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.581 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.581 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.582 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.583 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.583 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:39:30.583 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.583 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.583 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.583 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.583 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:39:30.584 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22580 2020-03-11T22:39:30.584 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.584 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.584 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.584 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6803/22580", 2020-03-11T22:39:30.584 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6805/22580", 2020-03-11T22:39:30.584 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6809/22580", 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6807/22580", 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.585 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:39:30.586 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "2a110545-206c-4222-a307-a9facc224c3b", 2020-03-11T22:39:30.586 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 0, 2020-03-11T22:39:30.586 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:39:30.586 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:39:30.586 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:39:30.586 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 0, 2020-03-11T22:39:30.586 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 0, 2020-03-11T22:39:30.587 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 10, 2020-03-11T22:39:30.587 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 22, 2020-03-11T22:39:30.587 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 30, 2020-03-11T22:39:30.587 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:39:30.587 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:39:30.587 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.587 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.588 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.588 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:30.588 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.588 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.588 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.588 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.588 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:30.589 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.589 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.589 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.589 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.589 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:39:30.589 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.590 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:30.591 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.591 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.591 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.591 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.591 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:30.591 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.591 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:30.592 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.593 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.593 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.593 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.593 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:30.593 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.593 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.593 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.594 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:30.594 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.594 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.594 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.594 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.594 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:30.595 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 18195 2020-03-11T22:39:30.595 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.595 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.595 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/18195", 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/18195", 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18195", 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18195", 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: "exists" 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.596 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.597 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.597 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:39:30.597 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "2971b697-7a4b-4c40-bd48-ff00e2071e11", 2020-03-11T22:39:30.597 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:39:30.597 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:39:30.597 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:39:30.597 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:39:30.598 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 15, 2020-03-11T22:39:30.598 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:39:30.598 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 35, 2020-03-11T22:39:30.598 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 0, 2020-03-11T22:39:30.598 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 31, 2020-03-11T22:39:30.598 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:39:30.598 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.599 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.600 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:30.600 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.600 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.600 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.600 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.600 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:39:30.600 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.601 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.601 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.601 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:30.601 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.601 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.601 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.601 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.602 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.603 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.603 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:30.603 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.603 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.603 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.603 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.603 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.604 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:39:30.605 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:30.605 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.605 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.605 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.605 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:39:30.605 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:30.606 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 22774 2020-03-11T22:39:30.606 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.606 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.606 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.606 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/22774", 2020-03-11T22:39:30.606 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/22774", 2020-03-11T22:39:30.606 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22774", 2020-03-11T22:39:30.607 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22774", 2020-03-11T22:39:30.607 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:39:30.607 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:39:30.607 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:39:30.607 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.607 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.607 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.608 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_xinfo": [ 2020-03-11T22:39:30.608 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.608 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:39:30.608 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:38:48.064710+0000", 2020-03-11T22:39:30.608 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:39:30.608 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:39:30.608 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:39:30.609 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:39:30.609 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:58.887011+0000", 2020-03-11T22:39:30.609 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:39:30.609 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.609 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.609 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:39:30.610 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:38:48.563581+0000", 2020-03-11T22:39:30.610 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:39:30.610 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:39:30.610 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:39:30.610 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:39:30.610 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:04.860801+0000", 2020-03-11T22:39:30.610 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:39:30.611 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.611 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.611 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:39:30.611 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:38:49.123406+0000", 2020-03-11T22:39:30.611 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:39:30.611 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:39:30.611 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:39:30.612 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:39:30.612 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:38:11.036763+0000", 2020-03-11T22:39:30.612 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:39:30.612 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.613 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.613 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap": [], 2020-03-11T22:39:30.613 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap_items": [], 2020-03-11T22:39:30.613 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_temp": [], 2020-03-11T22:39:30.614 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_temp": [], 2020-03-11T22:39:30.614 INFO:tasks.workunit.client.0.smithi142.stdout: "blacklist": {}, 2020-03-11T22:39:30.614 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profiles": { 2020-03-11T22:39:30.614 INFO:tasks.workunit.client.0.smithi142.stdout: "default": { 2020-03-11T22:39:30.614 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:39:30.615 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "2", 2020-03-11T22:39:30.615 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:39:30.615 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van" 2020-03-11T22:39:30.615 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.616 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.616 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps_queue": [], 2020-03-11T22:39:30.616 INFO:tasks.workunit.client.0.smithi142.stdout: "new_removed_snaps": [], 2020-03-11T22:39:30.616 INFO:tasks.workunit.client.0.smithi142.stdout: "new_purged_snaps": [], 2020-03-11T22:39:30.617 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_node_flags": {}, 2020-03-11T22:39:30.617 INFO:tasks.workunit.client.0.smithi142.stdout: "device_class_flags": {} 2020-03-11T22:39:30.617 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.617 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_metadata": [ 2020-03-11T22:39:30.618 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.618 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:39:30.618 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:39:30.618 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6804/22580,v1:127.0.0.1:6805/22580]", 2020-03-11T22:39:30.618 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:39:30.619 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:39:30.619 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:30.619 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:30.619 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:30.620 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:30.620 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:30.620 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:30.620 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:30.620 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:39:30.621 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:30.621 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:30.621 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:30.621 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:30.622 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:39:30.622 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:30.622 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:30.622 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:30.623 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:39:30.623 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:39:30.623 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:30.623 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:39:30.623 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:39:30.624 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:30.624 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:39:30.624 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6802/22580,v1:127.0.0.1:6803/22580]", 2020-03-11T22:39:30.624 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:39:30.625 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/22580,v1:127.0.0.1:6809/22580]", 2020-03-11T22:39:30.625 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/22580,v1:127.0.0.1:6807/22580]", 2020-03-11T22:39:30.625 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:39:30.625 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:39:30.626 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:30.626 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:30.626 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:30.626 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:39:30.627 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:30.627 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:30.627 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:39:30.627 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:39:30.628 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:30.628 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:39:30.628 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.628 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.628 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:39:30.629 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:39:30.629 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/18195,v1:127.0.0.1:6813/18195]", 2020-03-11T22:39:30.629 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:39:30.630 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:39:30.630 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:30.630 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:30.630 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:30.631 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:30.631 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:30.631 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:30.631 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:30.632 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:39:30.632 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:30.632 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:30.633 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:30.633 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:30.633 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:39:30.633 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:30.634 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:30.634 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:30.634 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:39:30.634 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:39:30.635 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:30.637 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:39:30.637 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:39:30.637 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:30.638 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:39:30.638 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/18195,v1:127.0.0.1:6811/18195]", 2020-03-11T22:39:30.638 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:39:30.638 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18195,v1:127.0.0.1:6817/18195]", 2020-03-11T22:39:30.639 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18195,v1:127.0.0.1:6815/18195]", 2020-03-11T22:39:30.639 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:39:30.639 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:39:30.639 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:30.639 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:30.640 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:30.640 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:39:30.640 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:30.640 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:30.641 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:39:30.642 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.642 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.642 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:39:30.642 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/22774,v1:127.0.0.1:6813/22774]", 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:39:30.643 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:30.644 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:39:30.645 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:30.646 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:30.647 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:30.648 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:39:30.649 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:39:30.649 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:30.649 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:39:30.649 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/22774,v1:127.0.0.1:6811/22774]", 2020-03-11T22:39:30.650 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:39:30.650 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22774,v1:127.0.0.1:6817/22774]", 2020-03-11T22:39:30.650 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22774,v1:127.0.0.1:6815/22774]", 2020-03-11T22:39:30.650 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:39:30.651 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:39:30.651 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:30.651 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:30.652 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:30.652 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:39:30.652 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:30.652 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:30.653 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:39:30.653 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:39:30.653 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:30.653 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:39:30.653 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.654 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.654 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_first_committed": 1, 2020-03-11T22:39:30.654 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_last_committed": 38, 2020-03-11T22:39:30.654 INFO:tasks.workunit.client.0.smithi142.stdout: "crushmap": { 2020-03-11T22:39:30.655 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": [ 2020-03-11T22:39:30.655 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.655 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:39:30.655 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.0", 2020-03-11T22:39:30.656 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:39:30.656 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.656 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.656 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:39:30.657 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.1", 2020-03-11T22:39:30.657 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:39:30.657 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.657 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.658 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:39:30.658 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.2", 2020-03-11T22:39:30.658 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:39:30.658 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.658 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.659 INFO:tasks.workunit.client.0.smithi142.stdout: "types": [ 2020-03-11T22:39:30.659 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.659 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 0, 2020-03-11T22:39:30.659 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd" 2020-03-11T22:39:30.659 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.659 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.659 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:39:30.660 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "host" 2020-03-11T22:39:30.660 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.660 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.660 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 2, 2020-03-11T22:39:30.660 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "chassis" 2020-03-11T22:39:30.660 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.660 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.661 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 3, 2020-03-11T22:39:30.661 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "rack" 2020-03-11T22:39:30.661 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.661 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.661 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 4, 2020-03-11T22:39:30.661 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "row" 2020-03-11T22:39:30.662 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.662 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.662 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 5, 2020-03-11T22:39:30.662 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pdu" 2020-03-11T22:39:30.662 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.662 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.662 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 6, 2020-03-11T22:39:30.663 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pod" 2020-03-11T22:39:30.663 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.663 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.663 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 7, 2020-03-11T22:39:30.663 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "room" 2020-03-11T22:39:30.663 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.664 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.664 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 8, 2020-03-11T22:39:30.664 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "datacenter" 2020-03-11T22:39:30.664 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.664 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.664 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 9, 2020-03-11T22:39:30.664 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "zone" 2020-03-11T22:39:30.665 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.665 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.665 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 10, 2020-03-11T22:39:30.665 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "region" 2020-03-11T22:39:30.665 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.665 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.666 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:39:30.666 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "root" 2020-03-11T22:39:30.666 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.666 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.666 INFO:tasks.workunit.client.0.smithi142.stdout: "buckets": [ 2020-03-11T22:39:30.666 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.666 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -1, 2020-03-11T22:39:30.667 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default", 2020-03-11T22:39:30.667 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:39:30.667 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:39:30.667 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:39:30.667 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:39:30.667 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:39:30.668 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:39:30.668 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.668 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:39:30.668 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:39:30.669 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:39:30.669 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.669 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.669 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.669 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.670 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -2, 2020-03-11T22:39:30.670 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default~hdd", 2020-03-11T22:39:30.670 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:39:30.670 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:39:30.671 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:39:30.671 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:39:30.671 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:39:30.672 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:39:30.672 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.672 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:39:30.672 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:39:30.673 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:39:30.673 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.673 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.674 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.674 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.674 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:39:30.674 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142", 2020-03-11T22:39:30.674 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:39:30.675 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:39:30.675 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:39:30.675 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:39:30.675 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:39:30.676 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:39:30.676 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.676 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:39:30.676 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:39:30.677 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:39:30.677 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.677 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.677 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:39:30.678 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:39:30.678 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:39:30.678 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.679 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.679 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:39:30.679 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:39:30.679 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:39:30.680 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.680 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.680 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.680 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.680 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:39:30.681 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142~hdd", 2020-03-11T22:39:30.681 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:39:30.681 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:39:30.681 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:39:30.682 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:39:30.682 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:39:30.682 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:39:30.682 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.682 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:39:30.683 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:39:30.683 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:39:30.683 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.683 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.684 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:39:30.684 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:39:30.684 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:39:30.684 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.684 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.685 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:39:30.685 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:39:30.685 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:39:30.685 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.686 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.686 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.686 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.686 INFO:tasks.workunit.client.0.smithi142.stdout: "rules": [ 2020-03-11T22:39:30.687 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.687 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 0, 2020-03-11T22:39:30.687 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "replicated_rule", 2020-03-11T22:39:30.687 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 0, 2020-03-11T22:39:30.687 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:39:30.688 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 1, 2020-03-11T22:39:30.688 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 10, 2020-03-11T22:39:30.688 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:39:30.688 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.689 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:39:30.689 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:39:30.689 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:39:30.690 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.690 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.690 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_firstn", 2020-03-11T22:39:30.690 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:39:30.691 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:39:30.691 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.691 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.691 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:39:30.691 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.692 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:39:30.692 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.692 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.692 INFO:tasks.workunit.client.0.smithi142.stdout: "tunables": { 2020-03-11T22:39:30.692 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_tries": 0, 2020-03-11T22:39:30.692 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:39:30.693 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_total_tries": 50, 2020-03-11T22:39:30.693 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:39:30.693 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:39:30.693 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_stable": 1, 2020-03-11T22:39:30.693 INFO:tasks.workunit.client.0.smithi142.stdout: "straw_calc_version": 1, 2020-03-11T22:39:30.693 INFO:tasks.workunit.client.0.smithi142.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:39:30.693 INFO:tasks.workunit.client.0.smithi142.stdout: "profile": "jewel", 2020-03-11T22:39:30.694 INFO:tasks.workunit.client.0.smithi142.stdout: "optimal_tunables": 1, 2020-03-11T22:39:30.694 INFO:tasks.workunit.client.0.smithi142.stdout: "legacy_tunables": 0, 2020-03-11T22:39:30.694 INFO:tasks.workunit.client.0.smithi142.stdout: "minimum_required_version": "jewel", 2020-03-11T22:39:30.694 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables": 1, 2020-03-11T22:39:30.694 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables2": 1, 2020-03-11T22:39:30.694 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v2_rules": 0, 2020-03-11T22:39:30.694 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables3": 1, 2020-03-11T22:39:30.695 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v3_rules": 0, 2020-03-11T22:39:30.695 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v4_buckets": 1, 2020-03-11T22:39:30.695 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables5": 1, 2020-03-11T22:39:30.695 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v5_rules": 0 2020-03-11T22:39:30.695 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.696 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_args": {} 2020-03-11T22:39:30.696 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.696 INFO:tasks.workunit.client.0.smithi142.stdout: "fsmap": { 2020-03-11T22:39:30.696 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:39:30.696 INFO:tasks.workunit.client.0.smithi142.stdout: "default_fscid": -1, 2020-03-11T22:39:30.697 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": { 2020-03-11T22:39:30.697 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": {}, 2020-03-11T22:39:30.697 INFO:tasks.workunit.client.0.smithi142.stdout: "ro_compat": {}, 2020-03-11T22:39:30.697 INFO:tasks.workunit.client.0.smithi142.stdout: "incompat": { 2020-03-11T22:39:30.697 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_1": "base v0.20", 2020-03-11T22:39:30.697 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:39:30.698 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:39:30.698 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:39:30.698 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:39:30.698 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:39:30.698 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_8": "no anchor table", 2020-03-11T22:39:30.698 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_9": "file layout v2", 2020-03-11T22:39:30.698 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:39:30.699 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.699 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.699 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_flags": { 2020-03-11T22:39:30.699 INFO:tasks.workunit.client.0.smithi142.stdout: "enable_multiple": false, 2020-03-11T22:39:30.699 INFO:tasks.workunit.client.0.smithi142.stdout: "ever_enabled_multiple": false 2020-03-11T22:39:30.699 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.699 INFO:tasks.workunit.client.0.smithi142.stdout: "standbys": [], 2020-03-11T22:39:30.700 INFO:tasks.workunit.client.0.smithi142.stdout: "filesystems": [] 2020-03-11T22:39:30.700 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.700 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:39:30.700 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:39:30.700 INFO:tasks.workunit.client.0.smithi142.stdout: "auth": { 2020-03-11T22:39:30.700 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:39:30.700 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 5, 2020-03-11T22:39:30.701 INFO:tasks.workunit.client.0.smithi142.stdout: "num_secrets": 3 2020-03-11T22:39:30.701 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.701 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:39:30.701 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_active": 1, 2020-03-11T22:39:30.701 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_unknown": 0, 2020-03-11T22:39:30.701 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osd": 3, 2020-03-11T22:39:30.701 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_sum": { 2020-03-11T22:39:30.702 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:39:30.702 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48300, 2020-03-11T22:39:30.702 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:39:30.702 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:39:30.702 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:39:30.702 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:30.702 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:39:30.703 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:30.703 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:30.703 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:30.703 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:30.703 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:39:30.703 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:39:30.703 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:39:30.704 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 100, 2020-03-11T22:39:30.704 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 100, 2020-03-11T22:39:30.704 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:30.704 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:30.704 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:30.704 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:30.704 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:30.705 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:30.705 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:39:30.705 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:30.705 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:30.705 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:39:30.705 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:39:30.705 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:39:30.706 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:39:30.706 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:39:30.706 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:30.706 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:30.706 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:30.706 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:30.706 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:30.707 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:30.707 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:30.707 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:30.707 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:30.707 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:39:30.707 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:39:30.707 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.708 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:39:30.708 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:39:30.708 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:39:30.708 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:39:30.708 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 0, 2020-03-11T22:39:30.708 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 0, 2020-03-11T22:39:30.708 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:39:30.709 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:30.709 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:39:30.709 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:39:30.709 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:39:30.709 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.709 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 16, 2020-03-11T22:39:30.709 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 16, 2020-03-11T22:39:30.710 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 2, 2020-03-11T22:39:30.710 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 2, 2020-03-11T22:39:30.710 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 0 2020-03-11T22:39:30.710 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.710 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum": { 2020-03-11T22:39:30.710 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:39:30.710 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:39:30.711 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:39:30.711 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:39:30.711 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:30.711 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:30.711 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:39:30.711 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3178880, 2020-03-11T22:39:30.711 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 32896, 2020-03-11T22:39:30.712 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 12, 2020-03-11T22:39:30.712 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145715, 2020-03-11T22:39:30.712 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393920, 2020-03-11T22:39:30.712 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:39:30.712 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:39:30.712 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318867374080, 2020-03-11T22:39:30.713 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:30.713 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 33685504, 2020-03-11T22:39:30.713 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 398171, 2020-03-11T22:39:30.713 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:39:30.713 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:30.713 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:39:30.713 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 12852, 2020-03-11T22:39:30.714 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221212620 2020-03-11T22:39:30.714 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.714 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:39:30.714 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:30.714 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:30.714 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 1, 2020-03-11T22:39:30.714 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:39:30.715 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:39:30.715 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:39:30.715 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.715 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:39:30.715 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:30.715 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:30.715 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:30.716 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:39:30.716 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.716 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:39:30.716 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:39:30.716 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.716 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum_by_class": { 2020-03-11T22:39:30.716 INFO:tasks.workunit.client.0.smithi142.stdout: "hdd": { 2020-03-11T22:39:30.717 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:39:30.717 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:39:30.717 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:39:30.717 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:39:30.717 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:30.717 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:30.717 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:39:30.718 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3178880, 2020-03-11T22:39:30.718 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 32896, 2020-03-11T22:39:30.718 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 12, 2020-03-11T22:39:30.718 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145715, 2020-03-11T22:39:30.718 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393920, 2020-03-11T22:39:30.718 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:39:30.718 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:39:30.719 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318867374080, 2020-03-11T22:39:30.719 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:30.719 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 33685504, 2020-03-11T22:39:30.719 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 398171, 2020-03-11T22:39:30.719 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:39:30.719 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:30.719 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:39:30.720 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 12852, 2020-03-11T22:39:30.720 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221212620 2020-03-11T22:39:30.720 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.720 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:39:30.720 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:30.720 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:30.720 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 1, 2020-03-11T22:39:30.721 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:39:30.721 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:39:30.721 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:39:30.721 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.721 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:39:30.721 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:30.722 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:30.722 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:30.722 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:39:30.722 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.722 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:39:30.722 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:39:30.722 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_stats": [ 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: "poolid": 1, 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48300, 2020-03-11T22:39:30.723 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:39:30.724 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:39:30.724 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:39:30.725 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:30.725 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:39:30.725 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:30.725 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:30.725 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:30.725 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:30.726 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:39:30.726 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:39:30.726 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:39:30.726 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 100, 2020-03-11T22:39:30.726 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 100, 2020-03-11T22:39:30.726 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:30.726 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:30.727 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:30.727 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:30.727 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:30.727 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:30.727 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:39:30.727 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:30.727 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:30.728 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:39:30.728 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:39:30.728 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:39:30.728 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:39:30.728 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:39:30.728 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:30.729 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:30.729 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:30.729 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:30.729 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:30.729 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:30.729 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:30.729 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:30.730 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:30.730 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:39:30.730 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:39:30.730 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.730 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:39:30.731 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:39:30.731 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:39:30.731 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:39:30.731 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 19660800, 2020-03-11T22:39:30.731 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 144900, 2020-03-11T22:39:30.731 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:39:30.731 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:30.732 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:39:30.732 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:39:30.732 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:39:30.732 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.732 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 16, 2020-03-11T22:39:30.732 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 16, 2020-03-11T22:39:30.733 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 2, 2020-03-11T22:39:30.733 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 2, 2020-03-11T22:39:30.733 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 3 2020-03-11T22:39:30.733 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.733 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.733 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_stats": [ 2020-03-11T22:39:30.734 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.734 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:39:30.734 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 146028888074 2020-03-11T22:39:30.734 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.734 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.734 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:39:30.735 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 42949672975 2020-03-11T22:39:30.735 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.735 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.735 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:39:30.735 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 150323855369 2020-03-11T22:39:30.736 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.736 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.736 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_state": [ 2020-03-11T22:39:30.736 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.736 INFO:tasks.workunit.client.0.smithi142.stdout: "state": "active+undersized+degraded", 2020-03-11T22:39:30.736 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 1 2020-03-11T22:39:30.736 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.737 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.737 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_osd": [ 2020-03-11T22:39:30.737 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.737 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:39:30.737 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:39:30.737 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:39:30.738 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:30.738 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.738 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.739 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:39:30.739 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:39:30.739 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 0, 2020-03-11T22:39:30.739 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:30.739 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.739 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.739 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:39:30.740 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 1, 2020-03-11T22:39:30.740 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:39:30.740 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:30.740 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.740 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.740 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": [ 2020-03-11T22:39:30.740 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:39:30.743 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:39:30.744 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": {} 2020-03-11T22:39:30.744 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.744 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:39:30.744 INFO:tasks.workunit.client.0.smithi142.stdout: "servicemap": { 2020-03-11T22:39:30.744 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:39:30.744 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:37:52.371998+0000", 2020-03-11T22:39:30.745 INFO:tasks.workunit.client.0.smithi142.stdout: "services": {} 2020-03-11T22:39:30.745 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:39:30.745 INFO:tasks.workunit.client.0.smithi142.stdout: "paxos": { 2020-03-11T22:39:30.745 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:39:30.745 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 305, 2020-03-11T22:39:30.745 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pn": 0, 2020-03-11T22:39:30.746 INFO:tasks.workunit.client.0.smithi142.stdout: "accepted_pn": 0 2020-03-11T22:39:30.746 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:39:30.746 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:39:30.747 INFO:tasks.workunit.client.0.smithi142.stderr:report 1150475218 2020-03-11T22:39:30.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:39:30.747 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:39:30.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:39:30.748 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:39:30.748 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:30.748 INFO:tasks.workunit.client.0.smithi142.stdout:writing non-divergent object existing_4 2020-03-11T22:39:30.847 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:30.848 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:56.009871+0000 28'100 38:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:30.848 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:30.848 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:30.849 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:30.857 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:39:30.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:39:30.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:30.930 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery 2020-03-11T22:39:31.252 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:31.252 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:56.009871+0000 28'100 38:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:31.252 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:31.252 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:31.253 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:31.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:39:31.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:39:31.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:39:31.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:39:31.263 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:31.263 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:31.263 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:39:31.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:39:31.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:39:31.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:39:31.354 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:31.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:39:31.361 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:39:31.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:39:31.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:39:31.362 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:31.362 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:31.362 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:39:31.362 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:39:31.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:39:31.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:39:31.453 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:31.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:39:31.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:31.460 INFO:tasks.workunit.client.0.smithi142.stdout:revive divergent 1 2020-03-11T22:39:31.779 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:31.779 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:56.009871+0000 28'100 38:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:31.779 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:31.779 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:31.781 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:31.789 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:39:32.406 INFO:tasks.workunit.client.0.smithi142.stderr:noup is set 2020-03-11T22:39:32.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:39:32.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:39:32.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:39:32.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:39:32.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:39:32.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:32.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:32.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:32.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:32.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:32.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:32.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:32.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:32.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:39:32.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:32.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:39:32.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:32.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:32.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:32.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:32.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:39:32.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:39:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:39:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:32.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:39:32.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:39:32.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:32.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:39:32.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:32.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:39:32.423 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:39:32.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:39:32.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:39:32.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:39:32.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:39:32.443 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:32.441+0000 7f6bcab04ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:32.445 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:32.442+0000 7f6bcab04ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:32.447 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:32.444+0000 7f6bcab04ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:32.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:39:32.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:39:32.962 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:32.959+0000 7f6bcab04ec0 -1 Falling back to public interface 2020-03-11T22:39:33.754 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:33.752+0000 7f6bcab04ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:39:37.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:39:37.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:37.770 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery divergent 2020-03-11T22:39:38.097 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:38.098 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:56.009871+0000 38'101 38:121 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:38.098 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:38.098 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:38.099 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:38.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:39:38.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:39:38.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:39:38.108 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:38.109 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:38.109 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:39:38.109 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:39:38.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:39:38.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:39:38.199 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:38.206 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:39:38.978 INFO:tasks.workunit.client.0.smithi142.stderr:noup is unset 2020-03-11T22:39:38.986 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:39:38.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:38.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:38.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:38.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:38.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:38.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:38.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:38.988 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:38.988 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:39:39.367 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 34 up_thru 37 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22580,v1:127.0.0.1:6803/22580] [v2:127.0.0.1:6804/22580,v1:127.0.0.1:6805/22580] exists,up 3cdb6518-067a-4954-96ef-4e52b4a57c3d 2020-03-11T22:39:39.367 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:39:39.367 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:39.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:39.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:39.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:39:39.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:39.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:39:39.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:39.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:39.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:39.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:39.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:39.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:39.709 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24242,v1:127.0.0.1:6819/24242] [v2:127.0.0.1:6820/24242,v1:127.0.0.1:6821/24242] exists,up 2a110545-206c-4222-a307-a9facc224c3b 2020-03-11T22:39:39.710 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:39:39.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:39.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:39.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:39.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:39:39.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:39.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:39:39.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:39.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:39.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:39.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:39.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:39.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:40.054 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22774,v1:127.0.0.1:6811/22774] [v2:127.0.0.1:6812/22774,v1:127.0.0.1:6813/22774] exists,up 2971b697-7a4b-4c40-bd48-ff00e2071e11 2020-03-11T22:39:40.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:40.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:40.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:40.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:40.376 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:40.376 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:56.009871+0000 38'101 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:40.376 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:40.376 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:40.377 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:40.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:39:40.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:40.386 INFO:tasks.workunit.client.0.smithi142.stdout:wait for peering 2020-03-11T22:39:40.705 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:40.706 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:56.009871+0000 38'101 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:40.706 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:40.706 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:40.707 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:40.715 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:39:52.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:39:52.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:52.040 INFO:tasks.workunit.client.0.smithi142.stdout:killing divergent 1 2020-03-11T22:39:52.371 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:52.371 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:39:39.219038+0000 38'101 44:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:52.371 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:52.372 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:52.372 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:52.380 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:39:52.381 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:39:52.381 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:39:52.381 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:39:52.381 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:39:52.382 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:39:52.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:39:52.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:39:52.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:39:52.486 INFO:tasks.workunit.client.0.smithi142.stdout:reviving divergent 1 2020-03-11T22:39:52.804 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:52.805 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:39:39.219038+0000 38'101 44:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:39:52.805 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:39:52.805 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:52.806 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:39:52.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:39:52.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:39:52.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:39:52.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:39:52.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:39:52.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:52.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:52.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:52.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:52.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:52.820 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:52.820 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:52.820 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:52.820 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:39:52.820 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:52.820 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:39:52.820 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:52.821 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:52.821 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:52.821 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:52.821 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:39:52.821 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:39:52.821 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:39:52.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:52.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:52.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:52.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:52.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:52.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:52.823 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:39:52.823 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:39:52.823 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:52.823 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:39:52.823 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:52.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:39:52.824 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:39:52.824 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:39:52.824 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:39:52.824 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:39:52.824 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:39:52.844 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:52.840+0000 7fe4a7d72ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:52.845 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:52.842+0000 7fe4a7d72ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:52.846 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:52.842+0000 7fe4a7d72ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:53.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:39:53.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:53.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:39:53.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:53.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:53.180 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:53.180 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:53.180 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:53.180 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:53.180 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:39:53.361 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:53.357+0000 7fe4a7d72ec0 -1 Falling back to public interface 2020-03-11T22:39:53.526 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24242,v1:127.0.0.1:6819/24242] [v2:127.0.0.1:6820/24242,v1:127.0.0.1:6821/24242] exists,up 2a110545-206c-4222-a307-a9facc224c3b 2020-03-11T22:39:53.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:53.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:53.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:53.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:39:54.274 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:39:54.270+0000 7fe4a7d72ec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-11T22:40:13.528 INFO:tasks.workunit.client.0.smithi142.stdout:allowing recovery 2020-03-11T22:40:13.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:40:13.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:40:13.857 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:13.857 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:39:55.313473+0000 44'102 47:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:40:13.857 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:40:13.857 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:13.858 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:40:13.866 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:40:13.867 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:40:13.988 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:13.995 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:40:13.995 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:40:14.116 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:14.123 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:40:14.123 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:40:14.244 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:14.251 INFO:tasks.workunit.client.0.smithi142.stdout:reading divergent objects 2020-03-11T22:40:14.251 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:40:14.251 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:40:14.569 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:14.570 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:39:55.313473+0000 44'102 47:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:38:13.340701+0000 0'0 2020-03-11T22:38:13.340701+0000 0 2020-03-11T22:40:14.570 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:40:14.570 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:14.571 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:40:14.579 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:40:14.581 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:40:14.581 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:14.582 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:40:14.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:14.628 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:40:14.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:14.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:40:14.683 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:14.684 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:40:14.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:40:14.713 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:14.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:14.717 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:38:36.771784+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:14.717 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:38:36.771784+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:14.717 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:14.718 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:40:14.718 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:40:14.718 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:40:14.718 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2020-03-11T22:40:14.718 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-03-11T22:38:36.771784+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:14.718 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:38.906+0000 7f6ba8cbf700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:40:14.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:14.719 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:14.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:40:14.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:40:14.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:40:14.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:40:14.723 INFO:tasks.workunit.client.0.smithi142.stdout:success 2020-03-11T22:40:15.195 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' does not exist 2020-03-11T22:40:15.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:40:15.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:15.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:15.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:15.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:15.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:15.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:15.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:40:15.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:40:15.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:40:15.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:40:15.312 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:15.312 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:15.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:15.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:15.313 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:15.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:15.316 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:40:15.317 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:40:15.317 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:40:15.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:40:15.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:40:15.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:40:15.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:15.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:40:15.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:40:15.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:15.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:40:15.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:40:15.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:40:15.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:40:15.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:15.333 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:15.333 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:40:15.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:40:15.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:40:15.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:40:15.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:40:15.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:40:15.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:40:15.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:40:15.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:40:15.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:40:15.336 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:15.336 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:15.337 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:15.337 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:15.337 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:15.338 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:15.338 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:40:15.338 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:40:15.338 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:40:15.339 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:40:15.339 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:40:15.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:40:15.340 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:15.340 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:40:15.340 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:40:15.341 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:15.341 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:40:15.342 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:40:15.342 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:40:15.343 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:40:15.343 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:15.343 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:15.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:40:15.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:40:15.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:40:15.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:40:15.345 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:40:15.345 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:15.346 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:15.346 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17403 2020-03-11T22:40:15.347 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:40:15.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:40:15.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:40:15.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:40:15.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:40:15.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:40:15.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:40:15.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:40:15.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:40:15.349 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:40:15.349 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:40:15.350 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:40:15.351 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:40:15.351 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:40:15.351 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:40:15.351 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:40:15.351 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:40:15.351 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:40:15.352 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:40:15.352 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:40:15.352 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:40:15.616 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:40:15.617 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:15.617 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:15.617 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:15.617 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:15.617 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:15.617 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:15.618 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:40:15.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:40:15.777 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:40:15.778 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:40:15.778 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:40:15.778 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:40:15.778 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:40:15.779 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:40:15.779 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:40:15.779 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:40:15.779 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:15.779 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:15.779 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:15.780 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:40:15.780 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:40:15.780 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get fsid 2020-03-11T22:40:15.864 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:40:15.864 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:40:15.865 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:40:15.865 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:40:15.865 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:40:15.865 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:40:15.865 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:40:15.865 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:40:15.865 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:15.866 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:15.866 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:15.866 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:40:15.866 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:40:15.866 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get mon_host 2020-03-11T22:40:15.950 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:40:15.950 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:40:15.951 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:40:15.951 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:40:15.951 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:40:15.951 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:40:15.951 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:40:16.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:40:16.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:16.826 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:16.826 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:16.826 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:16.826 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:16.826 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:16.827 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:40:16.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:40:16.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:40:16.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:40:16.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:16.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:16.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:40:16.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:16.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:40:16.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:16.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:16.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:16.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:16.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:40:16.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:40:16.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:16.857 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:16.857 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:16.857 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:16.857 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:16.857 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:16.857 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:16.858 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:16.858 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:16.858 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:16.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:16.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:16.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:16.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:16.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:16.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:16.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:16.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:16.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:16.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:16.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:16.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:40:16.860 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:16.861 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=99254390-63b0-48e6-be77-d00d55fcfdaa 2020-03-11T22:40:16.861 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 99254390-63b0-48e6-be77-d00d55fcfdaa' 2020-03-11T22:40:16.861 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:16.861 INFO:tasks.workunit.client.0.smithi142.stdout:add osd0 99254390-63b0-48e6-be77-d00d55fcfdaa 2020-03-11T22:40:16.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDQaGlez2bwMxAAxCpqDi6aVzfFBrg6Qj56mw== 2020-03-11T22:40:16.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDQaGlez2bwMxAAxCpqDi6aVzfFBrg6Qj56mw=="}' 2020-03-11T22:40:16.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 99254390-63b0-48e6-be77-d00d55fcfdaa -i td/divergent-priors/0/new.json 2020-03-11T22:40:17.206 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:40:17.216 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:40:17.217 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQaGlez2bwMxAAxCpqDi6aVzfFBrg6Qj56mw== --osd-uuid 99254390-63b0-48e6-be77-d00d55fcfdaa 2020-03-11T22:40:17.245 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:17.242+0000 7fa537c2cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:17.247 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:17.244+0000 7fa537c2cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:17.248 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:17.245+0000 7fa537c2cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:17.249 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:17.246+0000 7fa537c2cec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:17.249 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:17.246+0000 7fa537c2cec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:17.249 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:17.246+0000 7fa537c2cec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:17.308 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:17.305+0000 7fa537c2cec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:40:20.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:40:20.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:20.221 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd0 key to auth repository 2020-03-11T22:40:20.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:40:20.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:20.691 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:40:20.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:40:20.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:20.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:20.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:20.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:20.719 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:20.715+0000 7f7393669ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:20.721 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:20.717+0000 7f7393669ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:20.723 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:20.719+0000 7f7393669ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:21.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:40:21.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:21.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:21.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:21.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:21.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:21.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:21.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:21.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:21.043 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:40:21.238 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:21.235+0000 7f7393669ec0 -1 Falling back to public interface 2020-03-11T22:40:21.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:21.934 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:21.931+0000 7f7393669ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:40:22.392 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:22.392 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:22.392 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:22.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:22.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:22.393 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:40:22.757 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25549,v1:127.0.0.1:6803/25549] [v2:127.0.0.1:6804/25549,v1:127.0.0.1:6805/25549] exists,up 99254390-63b0-48e6-be77-d00d55fcfdaa 2020-03-11T22:40:22.757 INFO:tasks.workunit.client.0.smithi142.stdout:add osd1 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed 2020-03-11T22:40:22.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:22.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:22.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:22.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:40:22.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:40:22.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:22.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:22.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:40:22.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:22.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:40:22.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:22.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:22.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:22.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:22.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:40:22.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:40:22.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:22.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:22.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:22.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:22.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:22.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:22.768 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:22.768 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:22.768 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:22.768 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:22.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:22.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:22.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:22.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:22.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:22.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:22.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:22.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:22.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:22.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:22.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:22.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:40:22.770 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:22.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=277f3e48-aefe-4e39-9f54-e1f3bf3d05ed 2020-03-11T22:40:22.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed' 2020-03-11T22:40:22.770 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:22.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDWaGleYaIHLhAAvIW+FNg2XrBeIzCn9Dv2pg== 2020-03-11T22:40:22.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDWaGleYaIHLhAAvIW+FNg2XrBeIzCn9Dv2pg=="}' 2020-03-11T22:40:22.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed -i td/divergent-priors/1/new.json 2020-03-11T22:40:23.313 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:40:23.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:40:23.323 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDWaGleYaIHLhAAvIW+FNg2XrBeIzCn9Dv2pg== --osd-uuid 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed 2020-03-11T22:40:23.352 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:23.349+0000 7ffad798fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:23.354 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:23.351+0000 7ffad798fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:23.355 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:23.352+0000 7ffad798fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:23.356 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:23.353+0000 7ffad798fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:23.356 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:23.353+0000 7ffad798fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:23.356 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:23.353+0000 7ffad798fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:23.431 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:23.428+0000 7ffad798fec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:40:26.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:40:26.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:26.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:40:26.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:26.432 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd1 key to auth repository 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:40:26.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:26.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:26.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:26.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:26.914 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:26.911+0000 7fcaaf4e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:26.915 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:26.913+0000 7fcaaf4e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:26.917 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:26.914+0000 7fcaaf4e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:27.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:40:27.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:27.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:27.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:27.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:27.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:27.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:27.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:27.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:27.235 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:40:27.432 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:27.429+0000 7fcaaf4e6ec0 -1 Falling back to public interface 2020-03-11T22:40:27.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:28.188 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:28.185+0000 7fcaaf4e6ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:40:28.577 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:28.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:28.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:28.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:28.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:28.578 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:40:28.951 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25904,v1:127.0.0.1:6811/25904] [v2:127.0.0.1:6812/25904,v1:127.0.0.1:6813/25904] exists,up 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed 2020-03-11T22:40:28.951 INFO:tasks.workunit.client.0.smithi142.stdout:add osd2 1ebabf1d-927a-49d6-9158-8e49a3cc24b8 2020-03-11T22:40:28.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:28.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:28.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:28.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:40:28.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:40:28.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:28.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:28.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:40:28.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:28.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:40:28.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:28.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:28.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:28.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:28.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:40:28.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:40:28.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:28.956 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:28.956 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:28.956 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:28.956 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:28.956 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:28.957 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:28.957 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:28.957 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:28.957 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:28.957 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:40:28.957 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:28.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:28.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:28.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:28.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:28.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:28.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:28.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:28.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:28.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:28.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:40:28.962 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:28.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1ebabf1d-927a-49d6-9158-8e49a3cc24b8 2020-03-11T22:40:28.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 1ebabf1d-927a-49d6-9158-8e49a3cc24b8' 2020-03-11T22:40:28.962 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:28.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDcaGleuNpUORAAXL0QlWOxVSYoWVL7BkdJ5Q== 2020-03-11T22:40:28.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDcaGleuNpUORAAXL0QlWOxVSYoWVL7BkdJ5Q=="}' 2020-03-11T22:40:28.965 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1ebabf1d-927a-49d6-9158-8e49a3cc24b8 -i td/divergent-priors/2/new.json 2020-03-11T22:40:29.409 INFO:tasks.workunit.client.0.smithi142.stdout:2 2020-03-11T22:40:29.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:40:29.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDcaGleuNpUORAAXL0QlWOxVSYoWVL7BkdJ5Q== --osd-uuid 1ebabf1d-927a-49d6-9158-8e49a3cc24b8 2020-03-11T22:40:29.448 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:29.445+0000 7f6e3dbf3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:29.449 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:29.447+0000 7f6e3dbf3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:29.451 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:29.449+0000 7f6e3dbf3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:29.452 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:29.449+0000 7f6e3dbf3ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:29.452 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:29.449+0000 7f6e3dbf3ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:29.452 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:29.449+0000 7f6e3dbf3ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:29.511 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:29.509+0000 7f6e3dbf3ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:40:32.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:40:32.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:32.464 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd2 key to auth repository 2020-03-11T22:40:32.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:40:32.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:32.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:40:32.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:32.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:32.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:32.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:32.924 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:40:32.948 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:32.945+0000 7f4d26ac0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:32.950 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:32.947+0000 7f4d26ac0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:32.952 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:32.949+0000 7f4d26ac0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:33.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:40:33.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:33.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:33.270 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:40:33.467 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:33.464+0000 7f4d26ac0ec0 -1 Falling back to public interface 2020-03-11T22:40:33.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:34.267 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:40:34.265+0000 7f4d26ac0ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:40:34.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:34.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:34.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:34.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:34.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:34.615 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:40:34.982 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26259,v1:127.0.0.1:6819/26259] [v2:127.0.0.1:6820/26259,v1:127.0.0.1:6821/26259] exists,up 1ebabf1d-927a-49d6-9158-8e49a3cc24b8 2020-03-11T22:40:34.982 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:34.982 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:34.982 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:34.982 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:40:35.727 INFO:tasks.workunit.client.0.smithi142.stderr:noout is set 2020-03-11T22:40:35.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:40:36.379 INFO:tasks.workunit.client.0.smithi142.stderr:noin is set 2020-03-11T22:40:36.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:40:37.063 INFO:tasks.workunit.client.0.smithi142.stderr:nodown is set 2020-03-11T22:40:37.071 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:40:37.072 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:40:37.557 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' already exists 2020-03-11T22:40:37.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:40:38.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:40:39.201 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 size to 3 2020-03-11T22:40:39.209 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:40:39.877 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 min_size to 2 2020-03-11T22:40:39.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:40:39.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:39.886 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:40.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:40.231 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:40:40.232 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:40:40.232 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:40.232 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:40.232 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:40.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:40:40.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:40:40.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:40.364 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:40.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:40:40.491 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:40:40.491 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:40.491 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:40.619 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542147 2020-03-11T22:40:40.619 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-60129542147' 2020-03-11T22:40:40.619 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:40.620 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:40:40.620 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:40.621 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:40.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:40:40.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:40.622 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:40:40.622 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:40:40.622 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:40:40.623 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:40.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:40:40.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:41.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:41.977 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:42.331 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:40:42.331 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:42.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:40:42.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:42.332 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:42.333 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:40:42.333 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:42.334 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:40:42.334 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:40:42.334 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:42.334 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:40:42.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:40:42.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:42.677 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542147 2020-03-11T22:40:42.677 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:42.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:42.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542147 2020-03-11T22:40:42.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:42.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542147 2020-03-11T22:40:42.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2020-03-11T22:40:42.679 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:42.680 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 60129542147 2020-03-11T22:40:43.032 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542147 2020-03-11T22:40:43.032 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:40:43.032 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:43.032 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:43.032 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:43.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:43.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:40:43.033 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:43.033 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:43.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:43.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:43.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:43.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:43.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:43.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:43.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:43.124 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:43.465 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:43.466 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:40:43.466 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:40:43.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:43.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:43.466 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:43.594 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:40:43.594 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:40:43.595 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:43.595 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:43.723 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:40:43.773 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-03-11T22:40:43.774 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:43.774 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:43.850 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-11T22:40:43.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542150' 2020-03-11T22:40:43.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:43.851 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:40:43.851 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:43.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:43.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:40:43.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:43.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:40:43.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:40:43.859 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:43.860 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:40:44.213 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:40:44.213 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:44.214 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:40:44.214 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:44.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:44.216 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:40:44.216 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:44.216 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:40:44.217 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:40:44.217 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:44.217 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:40:44.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:40:44.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:44.560 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-11T22:40:44.560 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:44.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:44.561 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-11T22:40:44.561 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:44.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-11T22:40:44.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-11T22:40:44.562 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:44.563 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 60129542150 2020-03-11T22:40:44.925 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542150 -lt 60129542150 2020-03-11T22:40:44.925 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:44.926 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:44.926 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:45.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:40:45.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:45.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:45.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:45.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:45.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:45.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:45.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:45.716 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:40:45.717 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:45.717 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:45.717 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:46.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:40:46.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:40:46.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:40:46.191 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:40:46.191 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:40:46.511 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:40:46.520 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:40:46.520 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:40:46.520 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:46.520 INFO:tasks.workunit.client.0.smithi142.stdout:primary and soon to be divergent is 1 2020-03-11T22:40:46.840 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:46.841 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:40:38.058057+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:40:46.841 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:40:46.841 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:46.843 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:40:46.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:40:46.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:40:46.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:40:46.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:40:46.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:40:46.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:40:46.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:40:46.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:40:46.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:40:46.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:40:46.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:40:46.853 INFO:tasks.workunit.client.0.smithi142.stdout:writing initial objects 2020-03-11T22:40:46.853 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:40:46.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:46.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:40:46.966 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:46.966 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:40:47.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:40:47.048 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.048 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:40:47.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:40:47.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:40:47.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:40:47.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:40:47.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:40:47.382 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.382 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:40:47.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:40:47.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:40:47.502 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.502 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:40:47.582 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:40:47.623 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.623 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:40:47.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:40:47.698 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.699 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:40:47.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:40:47.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:40:47.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.857 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:40:47.896 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:40:47.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:47.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:40:48.015 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.015 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:40:48.052 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.052 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:40:48.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:40:48.217 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.218 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:40:48.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:40:48.297 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.297 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:40:48.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:40:48.373 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.374 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:40:48.459 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.459 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:40:48.498 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:40:48.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:40:48.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:40:48.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:40:48.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:40:48.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:40:48.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:40:48.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:40:48.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:40:48.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:48.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:40:49.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:40:49.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:40:49.147 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.147 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:40:49.227 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.227 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:40:49.331 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.332 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:40:49.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:40:49.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:40:49.511 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.511 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:40:49.552 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.552 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:40:49.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:40:49.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:40:49.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:40:49.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:40:49.912 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.912 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:40:49.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:40:49.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:49.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:40:50.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.031 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:40:50.113 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.113 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:40:50.188 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.188 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:40:50.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:40:50.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.303 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:40:50.341 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.341 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:40:50.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:40:50.461 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.461 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:40:50.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:40:50.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:40:50.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:40:50.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:40:50.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:40:50.789 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.789 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:40:50.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:40:50.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:40:50.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:50.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:40:51.056 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.056 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:40:51.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:40:51.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:40:51.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:40:51.256 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.256 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:40:51.295 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.295 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:40:51.333 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.334 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:40:51.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:40:51.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:40:51.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.495 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:40:51.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:40:51.573 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.573 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:40:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:40:51.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:40:51.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:40:51.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:40:51.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.865 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:40:51.903 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.903 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:40:51.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:51.942 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:40:52.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:52.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:40:52.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:52.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:40:52.192 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:52.192 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:40:52.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:52.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:40:52.376 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:40:52.435 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:40:52.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:40:52.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:52.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:52.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:52.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:52.437 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:40:52.437 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:52.437 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:52.437 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:52.437 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:52.438 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:52.492 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:52.492 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:52.492 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:52.492 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:52.493 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:52.835 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:52.836 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:40:52.836 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:40:52.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:52.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:52.836 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:52.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:40:52.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:40:52.999 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:52.999 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:53.128 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:40:53.128 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972' 2020-03-11T22:40:53.128 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:53.129 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:53.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542154 2020-03-11T22:40:53.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-42949672972 2-60129542154' 2020-03-11T22:40:53.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:53.259 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:40:53.259 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:53.259 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:53.260 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:40:53.260 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:53.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:40:53.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:40:53.261 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:53.261 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:40:53.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836492 2020-03-11T22:40:53.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:54.603 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:54.604 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:54.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836492 2020-03-11T22:40:54.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:54.954 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:40:54.955 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:54.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:54.956 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:40:54.956 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:54.957 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:40:54.957 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:40:54.957 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:54.958 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:40:55.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:40:55.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:55.305 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542154 2020-03-11T22:40:55.305 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:55.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:55.306 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542154 2020-03-11T22:40:55.307 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:55.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542154 2020-03-11T22:40:55.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2020-03-11T22:40:55.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:55.308 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 60129542154 2020-03-11T22:40:55.670 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542154 -lt 60129542154 2020-03-11T22:40:55.671 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:55.671 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:55.671 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:56.117 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:40:56.117 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:56.117 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:56.117 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:56.118 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:56.118 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:56.118 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:56.118 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:40:56.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:40:56.447 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:40:56.447 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:56.447 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:56.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:40:56.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:40:56.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:40:56.892 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:40:56.892 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:40:56.892 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:40:56.893 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:40:56.893 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:40:57.239 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:40:57.239 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:40:57.239 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:57.240 INFO:tasks.workunit.client.0.smithi142.stdout:blackholing osds 0 2 2020-03-11T22:40:57.601 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:57.601 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:40:38.058057+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:40:57.602 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:40:57.602 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:57.603 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:40:57.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:57.612 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:40:57.612 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:40:57.612 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:40:57.612 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:57.612 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:57.613 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:57.613 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:40:57.613 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:40:57.613 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:40:57.698 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:40:57.699 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:40:57.699 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:40:57.705 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:40:57.706 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:40:57.707 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:40:57.707 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:40:57.707 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:57.707 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:57.707 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:40:57.707 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:40:57.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:40:57.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:40:57.795 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:40:57.795 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:40:57.795 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:40:57.802 INFO:tasks.workunit.client.0.smithi142.stdout:create new divergent objects 2020-03-11T22:40:57.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:40:57.802 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:40:57.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:57.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:57.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:40:57.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:40:57.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:40:57.804 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:40:57.804 INFO:tasks.workunit.client.0.smithi142.stdout:writing divergent objects 2020-03-11T22:40:57.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:40:57.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:40:57.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:40:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:40:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:40:57.806 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:40:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:40:57.806 INFO:tasks.workunit.client.0.smithi142.stdout:remove divergent objects 2020-03-11T22:40:57.807 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:40:57.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:40:57.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:40:57.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:40:57.808 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:40:57.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:40:57.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:40:57.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:41:07.810 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:41:07.813 INFO:tasks.workunit.client.0.smithi142.stderr:rados: no process found 2020-03-11T22:41:07.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:41:07.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:07.814 INFO:tasks.workunit.client.0.smithi142.stdout:killing all the osds 2020-03-11T22:41:08.145 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:08.145 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:40:38.058057+0000 28'106 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:41:08.145 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:41:08.145 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:08.147 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:41:08.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:41:08.155 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:08.155 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:08.156 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:08.156 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:08.156 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:08.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:08.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:41:08.262 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:41:08.758 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already down. 2020-03-11T22:41:08.765 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:41:08.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:41:09.278 INFO:tasks.workunit.client.0.smithi142.stderr:osd.1 is already down. 2020-03-11T22:41:09.286 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:41:09.286 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:41:09.776 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already down. 2020-03-11T22:41:09.784 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:09.784 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:41:10.270 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already out. 2020-03-11T22:41:10.278 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:10.278 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:41:10.753 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already out. 2020-03-11T22:41:10.762 INFO:tasks.workunit.client.0.smithi142.stdout:bringing up non_divergent 0 2 2020-03-11T22:41:10.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:41:10.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:11.082 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:11.082 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:40:38.058057+0000 28'106 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:41:11.083 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:41:11.083 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:11.084 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:41:11.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:11.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:41:11.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:11.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:11.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:41:11.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:11.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:41:11.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:11.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:11.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:11.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:11.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:41:11.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:41:11.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:11.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:11.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:11.097 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:11.098 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:11.098 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:11.098 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:11.098 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:11.098 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:41:11.098 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:41:11.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:41:11.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:11.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:11.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:11.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:11.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:11.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:11.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:11.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:11.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:41:11.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:41:11.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:11.100 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:41:11.100 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:41:11.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:41:11.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:11.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:11.102 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:11.121 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:11.119+0000 7fabdf8bcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:11.123 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:11.120+0000 7fabdf8bcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:11.124 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:11.122+0000 7fabdf8bcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:11.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:41:11.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:11.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:11.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:11.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:11.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:11.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:11.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:11.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:11.442 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:41:11.641 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:11.638+0000 7fabdf8bcec0 -1 Falling back to public interface 2020-03-11T22:41:11.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:12.470 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:12.467+0000 7fabdf8bcec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:41:12.793 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:12.793 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:12.794 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:12.794 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:41:12.794 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:12.794 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:13.133 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30274,v1:127.0.0.1:6803/30274] [v2:127.0.0.1:6804/30274,v1:127.0.0.1:6805/30274] exists,up 99254390-63b0-48e6-be77-d00d55fcfdaa 2020-03-11T22:41:13.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:13.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:13.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:13.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:13.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:41:13.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:13.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:13.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:41:13.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:13.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:41:13.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:13.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:13.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:13.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:13.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:41:13.140 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:41:13.140 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:13.140 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:13.140 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:13.140 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:13.140 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:13.140 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:13.141 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:13.141 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:13.141 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:41:13.141 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:41:13.141 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:41:13.141 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:13.141 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:13.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:13.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:13.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:13.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:13.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:13.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:13.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:41:13.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:41:13.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:13.143 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:41:13.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:41:13.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:13.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:13.144 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:13.144 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:41:13.162 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:13.158+0000 7fc6b33fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:13.163 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:13.160+0000 7fc6b33fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:13.164 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:13.162+0000 7fc6b33fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:13.513 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:41:13.513 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:13.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:13.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:13.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:13.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:13.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:13.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:13.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:13.515 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:41:13.691 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:13.688+0000 7fc6b33fcec0 -1 Falling back to public interface 2020-03-11T22:41:13.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:14.574 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:41:14.571+0000 7fc6b33fcec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:41:14.861 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:14.862 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:14.862 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:14.862 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:14.862 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:14.862 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:41:15.202 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,28) [v2:127.0.0.1:6810/30468,v1:127.0.0.1:6811/30468] [v2:127.0.0.1:6812/30468,v1:127.0.0.1:6813/30468] exists,up 1ebabf1d-927a-49d6-9158-8e49a3cc24b8 2020-03-11T22:41:15.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:15.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:15.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:15.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:15.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:41:15.793 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already in. 2020-03-11T22:41:15.801 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:41:15.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:41:16.353 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already in. 2020-03-11T22:41:16.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:41:16.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:41:16.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:16.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:16.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:16.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:16.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:41:16.364 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:16.364 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:16.364 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:16.364 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:16.364 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:16.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:16.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:16.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:16.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:16.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:16.821 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:16.821 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:41:16.822 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:41:16.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:16.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:16.822 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:16.951 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:41:16.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:41:16.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:16.952 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:17.041 INFO:tasks.workunit.client.0.smithi142.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:41:17.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:41:17.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:41:17.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:17.050 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:17.178 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:41:17.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:41:17.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:17.179 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:41:17.180 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:17.180 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:17.180 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:41:17.181 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:17.181 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:41:17.182 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:41:17.182 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:17.182 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:41:17.521 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:41:17.521 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:18.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:18.523 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:18.867 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:41:18.867 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:18.868 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:41:18.868 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:18.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:18.869 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:41:18.869 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:18.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:41:18.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:41:18.870 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:18.871 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 2020-03-11T22:41:19.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:41:19.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:41:19.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:19.242 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:41:19.242 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:19.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:19.243 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:41:19.243 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:19.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:41:19.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:41:19.244 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:41:19.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:19.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:41:19.590 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:19.590 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:19.590 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:20.032 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:20.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:20.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:20.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:20.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:20.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:20.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:20.033 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:20.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:20.368 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:20.369 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:20.369 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:20.811 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:20.811 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:41:20.811 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:41:20.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:41:20.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:20.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:41:20.912 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:20.913 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:20.913 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:20.913 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:20.913 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:20.913 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:20.913 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:20.913 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:21.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:21.241 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:21.242 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:21.242 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:21.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:21.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:21.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:21.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:21.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:21.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:21.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:21.689 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:21.689 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:22.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:22.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:22.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:41:22.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:22.138 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:41:22.338 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:22.338 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:22.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:22.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:22.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:22.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:22.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:22.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:22.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:22.669 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:22.670 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:22.670 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:23.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:23.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:23.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:23.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:23.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:23.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:23.113 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:23.113 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:23.113 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:23.556 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:23.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:23.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:41:23.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:23.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:41:23.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:23.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:23.959 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:23.959 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:23.959 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:23.959 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:23.959 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:23.960 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:24.290 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:24.290 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:24.290 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:24.290 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:24.733 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:24.734 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:25.176 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:25.176 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:25.176 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:41:25.177 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:25.177 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:41:25.977 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:25.977 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:25.978 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:25.978 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:25.978 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:25.978 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:25.979 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:25.979 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:26.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:26.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:26.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:26.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:26.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:26.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:26.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:26.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:26.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:26.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:26.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:26.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:26.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:27.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:27.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:27.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:41:27.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:27.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:41:28.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:28.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:28.812 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:28.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:28.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:28.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:28.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:28.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:29.150 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:29.150 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:29.151 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:29.151 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:29.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:29.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:29.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:29.593 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:29.593 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:29.593 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:29.593 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:29.593 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:29.594 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:30.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:30.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:30.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:41:30.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:30.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:41:33.235 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:33.235 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:33.235 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:33.236 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:33.236 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:33.236 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:33.236 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:33.237 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:33.574 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:33.574 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:33.575 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:33.575 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:34.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:34.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:34.013 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:34.013 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:34.013 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:34.013 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:34.013 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:34.013 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:34.014 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:34.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:34.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:34.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:41:34.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:34.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:41:40.862 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:40.862 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:40.862 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:40.862 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:40.862 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:40.862 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:40.863 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:40.863 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:41.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:41.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:41.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:41.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:41.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:41.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:41.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:41.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:41.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:41.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:41.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:41.648 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:41.648 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:42.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:42.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:42.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:41:42.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:42.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:41:49.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:49.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:49.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:49.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:49.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:49.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:49.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:49.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:49.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:49.723 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:49.723 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:49.723 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:50.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:50.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:50.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:50.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:50.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:50.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:50.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:50.166 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:50.166 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:50.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:50.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:50.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:41:50.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:41:51.053 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:41:51.054 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_fingerprint": "4348c815-2b77-4a67-aa2b-f96e75ef9bf9", 2020-03-11T22:41:51.054 INFO:tasks.workunit.client.0.smithi142.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:41:51.054 INFO:tasks.workunit.client.0.smithi142.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:41:51.054 INFO:tasks.workunit.client.0.smithi142.stdout: "timestamp": "2020-03-11T22:41:51.050576+0000", 2020-03-11T22:41:51.055 INFO:tasks.workunit.client.0.smithi142.stdout: "tag": "", 2020-03-11T22:41:51.055 INFO:tasks.workunit.client.0.smithi142.stdout: "health": { 2020-03-11T22:41:51.055 INFO:tasks.workunit.client.0.smithi142.stdout: "status": "HEALTH_WARN", 2020-03-11T22:41:51.055 INFO:tasks.workunit.client.0.smithi142.stdout: "checks": { 2020-03-11T22:41:51.055 INFO:tasks.workunit.client.0.smithi142.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:41:51.055 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:51.055 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:41:51.056 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:41:51.056 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 17 2020-03-11T22:41:51.056 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.056 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [], 2020-03-11T22:41:51.056 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:41:51.056 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.057 INFO:tasks.workunit.client.0.smithi142.stdout: "OSD_DOWN": { 2020-03-11T22:41:51.057 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:51.057 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:41:51.057 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 osds down", 2020-03-11T22:41:51.057 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:41:51.057 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.057 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:41:51.058 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.058 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "osd.1 (root=default,host=smithi142) is down" 2020-03-11T22:41:51.058 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.058 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.058 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:41:51.058 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.058 INFO:tasks.workunit.client.0.smithi142.stdout: "PG_DEGRADED": { 2020-03-11T22:41:51.059 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:51.059 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:41:51.059 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:41:51.059 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:41:51.059 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.059 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:41:51.060 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.060 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:41:51.060 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.060 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.060 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:41:51.060 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.060 INFO:tasks.workunit.client.0.smithi142.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:41:51.061 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:41:51.061 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:41:51.061 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:41:51.061 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:41:51.061 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.061 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:41:51.062 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.062 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:41:51.062 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.062 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.062 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:41:51.062 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.063 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.063 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:41:51.063 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.063 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.063 INFO:tasks.workunit.client.0.smithi142.stdout: "mutes": [] 2020-03-11T22:41:51.063 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.063 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_first_committed": 1, 2020-03-11T22:41:51.064 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_last_committed": 1, 2020-03-11T22:41:51.064 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap": { 2020-03-11T22:41:51.064 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:41:51.064 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:41:51.064 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:40:15.368080Z", 2020-03-11T22:41:51.064 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:40:15.368080Z", 2020-03-11T22:41:51.065 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release": 15, 2020-03-11T22:41:51.065 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:41:51.065 INFO:tasks.workunit.client.0.smithi142.stdout: "features": { 2020-03-11T22:41:51.065 INFO:tasks.workunit.client.0.smithi142.stdout: "persistent": [ 2020-03-11T22:41:51.065 INFO:tasks.workunit.client.0.smithi142.stdout: "kraken", 2020-03-11T22:41:51.065 INFO:tasks.workunit.client.0.smithi142.stdout: "luminous", 2020-03-11T22:41:51.066 INFO:tasks.workunit.client.0.smithi142.stdout: "mimic", 2020-03-11T22:41:51.066 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap-prune", 2020-03-11T22:41:51.066 INFO:tasks.workunit.client.0.smithi142.stdout: "nautilus", 2020-03-11T22:41:51.066 INFO:tasks.workunit.client.0.smithi142.stdout: "octopus" 2020-03-11T22:41:51.066 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.066 INFO:tasks.workunit.client.0.smithi142.stdout: "optional": [] 2020-03-11T22:41:51.066 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.067 INFO:tasks.workunit.client.0.smithi142.stdout: "mons": [ 2020-03-11T22:41:51.067 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.067 INFO:tasks.workunit.client.0.smithi142.stdout: "rank": 0, 2020-03-11T22:41:51.067 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "a", 2020-03-11T22:41:51.067 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:41:51.067 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.068 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.068 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.068 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:41:51.068 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 0 2020-03-11T22:41:51.068 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.068 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.069 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.069 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:41:51.069 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:41:51.069 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 0, 2020-03-11T22:41:51.069 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 0 2020-03-11T22:41:51.069 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi142.stdout: "quorum": [ 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi142.stdout: 0 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.070 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap": { 2020-03-11T22:41:51.071 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 40, 2020-03-11T22:41:51.071 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:41:51.071 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:40:16.070548+0000", 2020-03-11T22:41:51.071 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:41:16.556767+0000", 2020-03-11T22:41:51.071 INFO:tasks.workunit.client.0.smithi142.stdout: "last_up_change": "2020-03-11T22:41:14.644622+0000", 2020-03-11T22:41:51.071 INFO:tasks.workunit.client.0.smithi142.stdout: "last_in_change": "2020-03-11T22:41:16.139140+0000", 2020-03-11T22:41:51.071 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:41:51.072 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_num": 5800384, 2020-03-11T22:41:51.072 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_set": [ 2020-03-11T22:41:51.072 INFO:tasks.workunit.client.0.smithi142.stdout: "nodown", 2020-03-11T22:41:51.072 INFO:tasks.workunit.client.0.smithi142.stdout: "noin", 2020-03-11T22:41:51.072 INFO:tasks.workunit.client.0.smithi142.stdout: "noout", 2020-03-11T22:41:51.072 INFO:tasks.workunit.client.0.smithi142.stdout: "pglog_hardlimit", 2020-03-11T22:41:51.073 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snapdirs", 2020-03-11T22:41:51.073 INFO:tasks.workunit.client.0.smithi142.stdout: "recovery_deletes", 2020-03-11T22:41:51.073 INFO:tasks.workunit.client.0.smithi142.stdout: "sortbitwise" 2020-03-11T22:41:51.073 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.073 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_version": 7, 2020-03-11T22:41:51.073 INFO:tasks.workunit.client.0.smithi142.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:41:51.074 INFO:tasks.workunit.client.0.smithi142.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:41:51.074 INFO:tasks.workunit.client.0.smithi142.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:41:51.074 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_snapshot": "", 2020-03-11T22:41:51.074 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_max": 1, 2020-03-11T22:41:51.074 INFO:tasks.workunit.client.0.smithi142.stdout: "max_osd": 3, 2020-03-11T22:41:51.074 INFO:tasks.workunit.client.0.smithi142.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:41:51.075 INFO:tasks.workunit.client.0.smithi142.stdout: "min_compat_client": "jewel", 2020-03-11T22:41:51.075 INFO:tasks.workunit.client.0.smithi142.stdout: "require_osd_release": "octopus", 2020-03-11T22:41:51.075 INFO:tasks.workunit.client.0.smithi142.stdout: "pools": [ 2020-03-11T22:41:51.075 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.075 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:41:51.075 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_name": "test", 2020-03-11T22:41:51.075 INFO:tasks.workunit.client.0.smithi142.stdout: "create_time": "2020-03-11T22:40:37.328602+0000", 2020-03-11T22:41:51.076 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": 1, 2020-03-11T22:41:51.076 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_names": "hashpspool", 2020-03-11T22:41:51.076 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:41:51.076 INFO:tasks.workunit.client.0.smithi142.stdout: "size": 3, 2020-03-11T22:41:51.076 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 2, 2020-03-11T22:41:51.076 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_rule": 0, 2020-03-11T22:41:51.076 INFO:tasks.workunit.client.0.smithi142.stdout: "object_hash": 2, 2020-03-11T22:41:51.077 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:41:51.077 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num": 1, 2020-03-11T22:41:51.077 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num": 1, 2020-03-11T22:41:51.077 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num_target": 1, 2020-03-11T22:41:51.077 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_target": 1, 2020-03-11T22:41:51.078 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_pending": 1, 2020-03-11T22:41:51.078 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pg_merge_meta": { 2020-03-11T22:41:51.078 INFO:tasks.workunit.client.0.smithi142.stdout: "source_pgid": "0.0", 2020-03-11T22:41:51.078 INFO:tasks.workunit.client.0.smithi142.stdout: "ready_epoch": 0, 2020-03-11T22:41:51.078 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_started": 0, 2020-03-11T22:41:51.079 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_clean": 0, 2020-03-11T22:41:51.079 INFO:tasks.workunit.client.0.smithi142.stdout: "source_version": "0'0", 2020-03-11T22:41:51.079 INFO:tasks.workunit.client.0.smithi142.stdout: "target_version": "0'0" 2020-03-11T22:41:51.079 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.079 INFO:tasks.workunit.client.0.smithi142.stdout: "last_change": "28", 2020-03-11T22:41:51.079 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend": "0", 2020-03-11T22:41:51.080 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:41:51.080 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:41:51.080 INFO:tasks.workunit.client.0.smithi142.stdout: "auid": 0, 2020-03-11T22:41:51.080 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:41:51.080 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_seq": 0, 2020-03-11T22:41:51.080 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_epoch": 0, 2020-03-11T22:41:51.080 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_snaps": [], 2020-03-11T22:41:51.081 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps": "[]", 2020-03-11T22:41:51.081 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_bytes": 0, 2020-03-11T22:41:51.081 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_objects": 0, 2020-03-11T22:41:51.081 INFO:tasks.workunit.client.0.smithi142.stdout: "tiers": [], 2020-03-11T22:41:51.081 INFO:tasks.workunit.client.0.smithi142.stdout: "tier_of": -1, 2020-03-11T22:41:51.081 INFO:tasks.workunit.client.0.smithi142.stdout: "read_tier": -1, 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi142.stdout: "write_tier": -1, 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_mode": "none", 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_bytes": 0, 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_objects": 0, 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:41:51.082 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:41:51.083 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:41:51.083 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_flush_age": 0, 2020-03-11T22:41:51.083 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_evict_age": 0, 2020-03-11T22:41:51.083 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profile": "", 2020-03-11T22:41:51.083 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_params": { 2020-03-11T22:41:51.084 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "none" 2020-03-11T22:41:51.084 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.084 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_period": 0, 2020-03-11T22:41:51.084 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_count": 0, 2020-03-11T22:41:51.084 INFO:tasks.workunit.client.0.smithi142.stdout: "use_gmt_hitset": true, 2020-03-11T22:41:51.085 INFO:tasks.workunit.client.0.smithi142.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:41:51.085 INFO:tasks.workunit.client.0.smithi142.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:41:51.085 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:41:51.085 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:41:51.085 INFO:tasks.workunit.client.0.smithi142.stdout: "grade_table": [], 2020-03-11T22:41:51.085 INFO:tasks.workunit.client.0.smithi142.stdout: "stripe_width": 0, 2020-03-11T22:41:51.086 INFO:tasks.workunit.client.0.smithi142.stdout: "expected_num_objects": 0, 2020-03-11T22:41:51.086 INFO:tasks.workunit.client.0.smithi142.stdout: "fast_read": false, 2020-03-11T22:41:51.086 INFO:tasks.workunit.client.0.smithi142.stdout: "options": {}, 2020-03-11T22:41:51.086 INFO:tasks.workunit.client.0.smithi142.stdout: "application_metadata": {} 2020-03-11T22:41:51.086 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.087 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.087 INFO:tasks.workunit.client.0.smithi142.stdout: "osds": [ 2020-03-11T22:41:51.087 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.087 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:41:51.088 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "99254390-63b0-48e6-be77-d00d55fcfdaa", 2020-03-11T22:41:51.088 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:41:51.088 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:41:51.088 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:41:51.089 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:41:51.089 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 5, 2020-03-11T22:41:51.089 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:41:51.090 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 34, 2020-03-11T22:41:51.090 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 39, 2020-03-11T22:41:51.090 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 29, 2020-03-11T22:41:51.091 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:41:51.091 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:41:51.091 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.091 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.092 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.092 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:41:51.092 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.092 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.093 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.093 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.093 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:41:51.093 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.093 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.093 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.094 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.094 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:41:51.094 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.094 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.094 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.094 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:41:51.095 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.095 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.095 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.095 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.095 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:41:51.096 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.096 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.096 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.096 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.097 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:51.097 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.097 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.097 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.097 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:41:51.098 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.098 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.098 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.098 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.098 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:41:51.099 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.099 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.099 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.099 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.100 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:51.101 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.102 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.102 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.102 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:41:51.102 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.103 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.103 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.103 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.103 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:41:51.104 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:41:51.104 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.104 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.104 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.104 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6803/30274", 2020-03-11T22:41:51.104 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6805/30274", 2020-03-11T22:41:51.105 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30274", 2020-03-11T22:41:51.105 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30274", 2020-03-11T22:41:51.106 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:41:51.106 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:41:51.106 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:41:51.106 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.106 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.106 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.107 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:41:51.107 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "277f3e48-aefe-4e39-9f54-e1f3bf3d05ed", 2020-03-11T22:41:51.107 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 0, 2020-03-11T22:41:51.107 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:41:51.107 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:41:51.107 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:41:51.108 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 0, 2020-03-11T22:41:51.108 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 0, 2020-03-11T22:41:51.108 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 10, 2020-03-11T22:41:51.108 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 22, 2020-03-11T22:41:51.108 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 30, 2020-03-11T22:41:51.108 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:41:51.108 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:41:51.109 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.109 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.109 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.109 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:51.109 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.110 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.110 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.110 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.111 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:51.113 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.114 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.114 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.115 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.115 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:41:51.115 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.115 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.115 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.115 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:51.116 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.116 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.116 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.116 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.116 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:51.116 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.117 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.117 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.117 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.117 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:51.117 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.118 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.118 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.118 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:51.118 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.118 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.118 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.119 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.119 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:51.119 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.119 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.119 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.119 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.119 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.120 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.121 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.121 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:51.121 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:41:51.121 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.121 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.121 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.121 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/25904", 2020-03-11T22:41:51.122 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/25904", 2020-03-11T22:41:51.122 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25904", 2020-03-11T22:41:51.122 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25904", 2020-03-11T22:41:51.122 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:41:51.122 INFO:tasks.workunit.client.0.smithi142.stdout: "exists" 2020-03-11T22:41:51.122 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.122 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "1ebabf1d-927a-49d6-9158-8e49a3cc24b8", 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:41:51.123 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 14, 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 35, 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 0, 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 31, 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.124 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.125 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.125 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:41:51.125 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.125 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.125 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.125 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.125 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:41:51.126 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.126 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.126 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.126 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.126 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:41:51.126 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.126 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:41:51.127 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.128 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.129 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:41:51.130 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.131 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.131 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.131 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:41:51.131 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:41:51.131 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:41:51.131 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.131 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/30468", 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/30468", 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30468", 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30468", 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:41:51.132 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:41:51.133 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.133 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.133 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.133 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_xinfo": [ 2020-03-11T22:41:51.133 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:41:08.565604+0000", 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:22.946554+0000", 2020-03-11T22:41:51.134 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:41:51.135 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.135 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.135 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:41:51.135 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:41:09.078441+0000", 2020-03-11T22:41:51.135 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:41:51.135 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:41:51.135 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:29.211602+0000", 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:41:51.136 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:41:09.599535+0000", 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:35.311405+0000", 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:41:51.137 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap": [], 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap_items": [], 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_temp": [], 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_temp": [], 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: "blacklist": {}, 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profiles": { 2020-03-11T22:41:51.138 INFO:tasks.workunit.client.0.smithi142.stdout: "default": { 2020-03-11T22:41:51.139 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:41:51.139 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "2", 2020-03-11T22:41:51.139 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:41:51.139 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van" 2020-03-11T22:41:51.139 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.139 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.139 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps_queue": [], 2020-03-11T22:41:51.140 INFO:tasks.workunit.client.0.smithi142.stdout: "new_removed_snaps": [], 2020-03-11T22:41:51.140 INFO:tasks.workunit.client.0.smithi142.stdout: "new_purged_snaps": [], 2020-03-11T22:41:51.140 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_node_flags": {}, 2020-03-11T22:41:51.140 INFO:tasks.workunit.client.0.smithi142.stdout: "device_class_flags": {} 2020-03-11T22:41:51.140 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.140 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_metadata": [ 2020-03-11T22:41:51.140 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.141 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:41:51.141 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:41:51.141 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6804/30274,v1:127.0.0.1:6805/30274]", 2020-03-11T22:41:51.141 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:41:51.141 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:41:51.141 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:51.142 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:51.142 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:51.142 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:51.142 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:51.142 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:51.142 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:51.142 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:51.143 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:51.144 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:41:51.144 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:41:51.144 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:51.144 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:41:51.144 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:41:51.144 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:51.144 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6802/30274,v1:127.0.0.1:6803/30274]", 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30274,v1:127.0.0.1:6809/30274]", 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30274,v1:127.0.0.1:6807/30274]", 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:41:51.145 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:41:51.146 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:51.146 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:51.146 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:51.146 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:41:51.146 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:51.146 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:51.146 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:41:51.147 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:41:51.147 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:51.147 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:41:51.147 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.147 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.147 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:41:51.147 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/25904,v1:127.0.0.1:6813/25904]", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:51.148 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:51.149 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:51.149 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:51.149 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:41:51.149 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:51.149 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:51.149 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:51.149 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:51.150 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/25904,v1:127.0.0.1:6811/25904]", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25904,v1:127.0.0.1:6817/25904]", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25904,v1:127.0.0.1:6815/25904]", 2020-03-11T22:41:51.151 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:41:51.152 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:41:51.152 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:51.152 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:51.152 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:51.152 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:41:51.152 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:51.152 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:41:51.153 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:41:51.154 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/30468,v1:127.0.0.1:6813/30468]", 2020-03-11T22:41:51.154 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:41:51.154 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:41:51.154 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:41:51.154 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:41:51.154 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:41:51.154 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:41:51.155 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:41:51.155 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:41:51.155 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:41:51.155 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:41:51.155 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:41:51.155 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:41:51.155 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:41:51.156 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:41:51.157 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:41:51.157 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:41:51.157 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:41:51.157 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:41:51.157 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/30468,v1:127.0.0.1:6811/30468]", 2020-03-11T22:41:51.157 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:41:51.157 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30468,v1:127.0.0.1:6817/30468]", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30468,v1:127.0.0.1:6815/30468]", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:41:51.158 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.159 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_first_committed": 1, 2020-03-11T22:41:51.160 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_last_committed": 40, 2020-03-11T22:41:51.160 INFO:tasks.workunit.client.0.smithi142.stdout: "crushmap": { 2020-03-11T22:41:51.160 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": [ 2020-03-11T22:41:51.160 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.160 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:41:51.160 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.0", 2020-03-11T22:41:51.160 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.1", 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.161 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:41:51.162 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.2", 2020-03-11T22:41:51.162 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:41:51.162 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.162 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.162 INFO:tasks.workunit.client.0.smithi142.stdout: "types": [ 2020-03-11T22:41:51.162 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.162 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 0, 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd" 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "host" 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.163 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 2, 2020-03-11T22:41:51.164 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "chassis" 2020-03-11T22:41:51.164 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.164 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.164 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 3, 2020-03-11T22:41:51.164 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "rack" 2020-03-11T22:41:51.164 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.164 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 4, 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "row" 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 5, 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pdu" 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.165 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 6, 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pod" 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 7, 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "room" 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.166 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.167 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 8, 2020-03-11T22:41:51.167 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "datacenter" 2020-03-11T22:41:51.167 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.167 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.167 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 9, 2020-03-11T22:41:51.167 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "zone" 2020-03-11T22:41:51.167 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.168 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.168 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 10, 2020-03-11T22:41:51.168 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "region" 2020-03-11T22:41:51.168 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.168 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.168 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:41:51.168 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "root" 2020-03-11T22:41:51.169 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.169 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.169 INFO:tasks.workunit.client.0.smithi142.stdout: "buckets": [ 2020-03-11T22:41:51.169 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.169 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -1, 2020-03-11T22:41:51.169 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default", 2020-03-11T22:41:51.169 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:41:51.170 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:41:51.170 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:41:51.170 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:41:51.170 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:41:51.170 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:41:51.170 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.170 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -2, 2020-03-11T22:41:51.171 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default~hdd", 2020-03-11T22:41:51.172 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:41:51.172 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:41:51.172 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:41:51.172 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:41:51.172 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:41:51.172 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:41:51.172 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.173 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:41:51.173 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:41:51.173 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:41:51.173 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.173 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.173 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.173 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142", 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:41:51.174 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:41:51.175 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.175 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:41:51.175 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:41:51.175 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:41:51.175 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.175 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.175 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:41:51.176 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.177 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.177 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.177 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.177 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:41:51.177 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142~hdd", 2020-03-11T22:41:51.177 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:41:51.177 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:41:51.178 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.179 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:41:51.180 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:41:51.180 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:41:51.180 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.180 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.180 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.180 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.180 INFO:tasks.workunit.client.0.smithi142.stdout: "rules": [ 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 0, 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "replicated_rule", 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 0, 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 1, 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 10, 2020-03-11T22:41:51.181 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:41:51.182 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.182 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:41:51.182 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:41:51.182 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:41:51.182 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.182 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.182 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_firstn", 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:41:51.183 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: "tunables": { 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_tries": 0, 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_total_tries": 50, 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:41:51.184 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_stable": 1, 2020-03-11T22:41:51.185 INFO:tasks.workunit.client.0.smithi142.stdout: "straw_calc_version": 1, 2020-03-11T22:41:51.185 INFO:tasks.workunit.client.0.smithi142.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:41:51.185 INFO:tasks.workunit.client.0.smithi142.stdout: "profile": "jewel", 2020-03-11T22:41:51.185 INFO:tasks.workunit.client.0.smithi142.stdout: "optimal_tunables": 1, 2020-03-11T22:41:51.185 INFO:tasks.workunit.client.0.smithi142.stdout: "legacy_tunables": 0, 2020-03-11T22:41:51.185 INFO:tasks.workunit.client.0.smithi142.stdout: "minimum_required_version": "jewel", 2020-03-11T22:41:51.185 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables": 1, 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables2": 1, 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v2_rules": 0, 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables3": 1, 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v3_rules": 0, 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v4_buckets": 1, 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables5": 1, 2020-03-11T22:41:51.186 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v5_rules": 0 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_args": {} 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: "fsmap": { 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: "default_fscid": -1, 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": { 2020-03-11T22:41:51.187 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": {}, 2020-03-11T22:41:51.188 INFO:tasks.workunit.client.0.smithi142.stdout: "ro_compat": {}, 2020-03-11T22:41:51.188 INFO:tasks.workunit.client.0.smithi142.stdout: "incompat": { 2020-03-11T22:41:51.188 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_1": "base v0.20", 2020-03-11T22:41:51.188 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:41:51.188 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:41:51.188 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:41:51.188 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:41:51.189 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:41:51.189 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_8": "no anchor table", 2020-03-11T22:41:51.189 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_9": "file layout v2", 2020-03-11T22:41:51.189 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:41:51.189 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.189 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.189 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_flags": { 2020-03-11T22:41:51.190 INFO:tasks.workunit.client.0.smithi142.stdout: "enable_multiple": false, 2020-03-11T22:41:51.190 INFO:tasks.workunit.client.0.smithi142.stdout: "ever_enabled_multiple": false 2020-03-11T22:41:51.190 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.190 INFO:tasks.workunit.client.0.smithi142.stdout: "standbys": [], 2020-03-11T22:41:51.190 INFO:tasks.workunit.client.0.smithi142.stdout: "filesystems": [] 2020-03-11T22:41:51.190 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.190 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:41:51.191 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:41:51.191 INFO:tasks.workunit.client.0.smithi142.stdout: "auth": { 2020-03-11T22:41:51.191 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:41:51.191 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 5, 2020-03-11T22:41:51.191 INFO:tasks.workunit.client.0.smithi142.stdout: "num_secrets": 3 2020-03-11T22:41:51.192 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.192 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:41:51.192 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_active": 1, 2020-03-11T22:41:51.192 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_unknown": 0, 2020-03-11T22:41:51.192 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osd": 3, 2020-03-11T22:41:51.192 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_sum": { 2020-03-11T22:41:51.193 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:41:51.193 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48300, 2020-03-11T22:41:51.193 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:41:51.194 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:41:51.194 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:41:51.194 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:51.194 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:41:51.194 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:51.194 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:51.194 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 100, 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 100, 2020-03-11T22:41:51.195 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:51.196 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:51.196 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:51.196 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:51.196 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:51.196 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:51.196 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:41:51.197 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:51.197 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:51.197 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:41:51.197 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:41:51.197 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:41:51.197 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:41:51.197 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:41:51.198 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:51.198 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:51.198 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:51.198 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:51.199 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:51.200 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:51.200 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:51.200 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:51.200 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:51.201 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:41:51.202 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:41:51.202 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.202 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:41:51.202 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:41:51.202 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:41:51.202 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:41:51.203 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 0, 2020-03-11T22:41:51.203 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 0, 2020-03-11T22:41:51.203 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:41:51.203 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:51.203 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:41:51.203 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:41:51.203 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:41:51.204 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.204 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 16, 2020-03-11T22:41:51.204 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 16, 2020-03-11T22:41:51.204 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 2, 2020-03-11T22:41:51.204 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 2, 2020-03-11T22:41:51.204 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 0 2020-03-11T22:41:51.204 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.205 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum": { 2020-03-11T22:41:51.205 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:41:51.205 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:41:51.205 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:41:51.205 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:41:51.205 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:51.205 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:51.206 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:41:51.206 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3179328, 2020-03-11T22:41:51.206 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 33280, 2020-03-11T22:41:51.206 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 14, 2020-03-11T22:41:51.206 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145713, 2020-03-11T22:41:51.206 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393472, 2020-03-11T22:41:51.206 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:41:51.207 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:41:51.207 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318866915328, 2020-03-11T22:41:51.207 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:51.207 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 34078720, 2020-03-11T22:41:51.207 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 408502, 2020-03-11T22:41:51.207 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:41:51.207 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 14420, 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221211052 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:51.208 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:51.209 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:41:51.209 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:41:51.209 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:41:51.209 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.209 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:41:51.209 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:51.209 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:51.210 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:51.210 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:41:51.210 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.210 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:41:51.210 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:41:51.210 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.210 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum_by_class": { 2020-03-11T22:41:51.211 INFO:tasks.workunit.client.0.smithi142.stdout: "hdd": { 2020-03-11T22:41:51.211 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:41:51.211 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:41:51.211 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:41:51.211 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:41:51.211 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:41:51.211 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:41:51.212 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:41:51.212 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3179328, 2020-03-11T22:41:51.212 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 33280, 2020-03-11T22:41:51.212 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 14, 2020-03-11T22:41:51.212 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145713, 2020-03-11T22:41:51.212 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393472, 2020-03-11T22:41:51.212 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:41:51.213 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:41:51.213 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318866915328, 2020-03-11T22:41:51.213 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:41:51.213 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 34078720, 2020-03-11T22:41:51.213 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 408502, 2020-03-11T22:41:51.213 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:41:51.213 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:51.214 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:41:51.214 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 14420, 2020-03-11T22:41:51.214 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221211052 2020-03-11T22:41:51.214 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.214 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:41:51.214 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:41:51.214 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:41:51.215 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:41:51.215 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:41:51.215 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:41:51.215 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:41:51.215 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.215 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:41:51.215 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.216 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_stats": [ 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: "poolid": 1, 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48300, 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:41:51.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:41:51.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:41:51.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:41:51.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:41:51.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:41:51.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:41:51.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:41:51.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 100, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 100, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:41:51.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:41:51.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:41:51.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:41:51.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:41:51.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:41:51.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:41:51.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:41:51.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:41:51.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:41:51.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:41:51.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:41:51.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:41:51.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:41:51.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:41:51.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:41:51.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 19529728, 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 143228, 2020-03-11T22:41:51.223 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 16, 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 16, 2020-03-11T22:41:51.224 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 2, 2020-03-11T22:41:51.225 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 2, 2020-03-11T22:41:51.225 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 3 2020-03-11T22:41:51.225 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.225 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.225 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_stats": [ 2020-03-11T22:41:51.225 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.225 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 146028888074 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 42949672975 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.226 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:41:51.227 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 150323855370 2020-03-11T22:41:51.227 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.227 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.227 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_state": [ 2020-03-11T22:41:51.227 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.227 INFO:tasks.workunit.client.0.smithi142.stdout: "state": "active+undersized+degraded", 2020-03-11T22:41:51.227 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 1 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_osd": [ 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:41:51.228 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 0, 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.229 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.230 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:41:51.230 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 1, 2020-03-11T22:41:51.230 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:41:51.230 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:41:51.230 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.230 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.230 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": [ 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": {} 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: "servicemap": { 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:41:51.231 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:40:16.070967+0000", 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: "services": {} 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: "paxos": { 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 301, 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pn": 0, 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: "accepted_pn": 0 2020-03-11T22:41:51.232 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:41:51.233 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:41:51.234 INFO:tasks.workunit.client.0.smithi142.stderr:report 1488782382 2020-03-11T22:41:51.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:41:51.234 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:41:51.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:41:51.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:41:51.234 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:41:51.235 INFO:tasks.workunit.client.0.smithi142.stdout:writing non-divergent object existing_4 2020-03-11T22:41:51.384 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:51.384 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:32.005802+0000 28'100 40:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:41:51.384 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:41:51.384 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:51.385 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:41:51.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:41:51.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:41:51.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:41:51.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:51.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:51.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:51.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:51.440 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:41:51.441 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:51.441 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:51.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:51.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:51.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:51.496 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:51.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:51.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:51.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:51.497 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:51.838 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:51.839 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:41:51.839 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:41:51.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:51.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:51.839 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:51.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888076 2020-03-11T22:41:51.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076' 2020-03-11T22:41:51.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:51.968 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:52.058 INFO:tasks.workunit.client.0.smithi142.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076 1-' 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:52.066 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:52.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:41:52.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076 1- 2-150323855372' 2020-03-11T22:41:52.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:52.212 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888076 2020-03-11T22:41:52.212 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:52.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:52.213 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888076 2020-03-11T22:41:52.213 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:52.214 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888076 2020-03-11T22:41:52.214 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888076' 2020-03-11T22:41:52.214 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:52.214 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 146028888076 2020-03-11T22:41:52.556 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888076 -lt 146028888076 2020-03-11T22:41:52.556 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:52.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:41:52.557 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:52.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:52.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:41:52.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:52.558 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:41:52.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:41:52.559 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 2020-03-11T22:41:52.559 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:52.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672975 -lt 2020-03-11T22:41:52.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672975: unary operator expected 2020-03-11T22:41:52.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:52.906 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855372 2020-03-11T22:41:52.906 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:52.906 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:52.907 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855372 2020-03-11T22:41:52.907 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:52.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:41:52.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2020-03-11T22:41:52.908 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:52.908 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 150323855372 2020-03-11T22:41:53.251 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855370 -lt 150323855372 2020-03-11T22:41:53.251 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:54.252 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:54.252 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:54.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855372 -lt 150323855372 2020-03-11T22:41:54.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:54.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:54.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:55.052 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:55.053 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:55.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:55.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:55.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:55.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:55.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:55.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:55.389 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:55.389 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:55.390 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:55.390 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:55.835 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:55.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:41:55.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:41:55.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:41:55.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:55.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:41:55.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:55.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:55.937 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:55.937 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:55.937 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:55.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:55.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:55.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:56.267 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:56.268 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:56.268 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:56.268 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:56.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:56.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:56.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:56.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:56.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:56.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:56.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:56.711 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:56.711 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:57.152 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:57.152 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:57.152 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:41:57.152 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:57.153 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:41:57.353 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:57.353 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:57.354 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:57.354 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:57.354 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:57.354 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:57.354 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:57.354 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:57.683 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:57.684 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:57.684 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:57.684 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:58.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:58.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:58.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:58.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:58.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:58.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:58.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:58.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:58.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:58.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:58.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:58.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:41:58.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:58.573 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:41:58.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:58.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:58.974 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:58.974 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:58.974 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:58.974 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:58.974 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:58.974 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:59.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:59.302 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:59.302 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:59.303 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:59.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:59.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:59.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:59.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:59.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:59.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:59.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:59.746 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:59.746 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:42:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:42:00.989 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:00.989 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:00.990 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:00.990 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:00.991 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:00.991 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:00.991 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:00.991 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:01.326 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:01.327 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:01.327 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:01.327 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:01.771 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:01.771 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:01.771 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:01.771 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:01.771 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:01.772 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:01.772 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:01.772 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:01.772 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:02.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:02.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:02.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:42:02.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:02.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:42:03.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:03.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:03.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:03.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:03.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:03.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:03.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:03.814 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:04.153 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:04.154 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:04.154 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:04.154 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:04.600 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:04.600 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:04.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:04.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:04.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:04.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:04.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:04.601 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:04.601 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:05.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:05.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:05.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:42:05.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:05.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:42:08.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:08.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:08.247 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:08.247 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:08.248 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:08.248 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:08.248 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:08.248 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:08.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:08.586 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:08.586 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:08.587 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:09.034 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:09.035 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:09.035 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:09.035 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:09.035 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:09.035 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:09.035 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:09.035 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:09.036 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:09.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:09.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:09.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:42:09.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:09.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:42:15.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:15.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:15.887 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:15.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:15.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:15.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:15.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:15.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:16.227 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:16.227 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:16.227 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:16.228 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:16.674 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:16.674 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:16.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:16.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:16.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:16.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:16.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:16.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:16.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:17.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:17.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:17.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:42:17.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:17.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:42:24.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:24.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:24.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:24.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:24.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:24.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:24.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:24.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:24.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:24.759 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:24.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:24.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:25.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:42:25.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:25.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:25.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:25.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:25.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:25.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:25.204 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:25.205 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:25.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:25.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:25.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:42:25.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:42:26.086 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:42:26.086 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_fingerprint": "4348c815-2b77-4a67-aa2b-f96e75ef9bf9", 2020-03-11T22:42:26.087 INFO:tasks.workunit.client.0.smithi142.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:42:26.087 INFO:tasks.workunit.client.0.smithi142.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:42:26.087 INFO:tasks.workunit.client.0.smithi142.stdout: "timestamp": "2020-03-11T22:42:26.083183+0000", 2020-03-11T22:42:26.087 INFO:tasks.workunit.client.0.smithi142.stdout: "tag": "", 2020-03-11T22:42:26.087 INFO:tasks.workunit.client.0.smithi142.stdout: "health": { 2020-03-11T22:42:26.087 INFO:tasks.workunit.client.0.smithi142.stdout: "status": "HEALTH_WARN", 2020-03-11T22:42:26.087 INFO:tasks.workunit.client.0.smithi142.stdout: "checks": { 2020-03-11T22:42:26.088 INFO:tasks.workunit.client.0.smithi142.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:42:26.088 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:26.088 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:42:26.088 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:42:26.088 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 17 2020-03-11T22:42:26.088 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.088 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [], 2020-03-11T22:42:26.089 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:42:26.089 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.089 INFO:tasks.workunit.client.0.smithi142.stdout: "OSD_DOWN": { 2020-03-11T22:42:26.089 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:26.089 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:42:26.089 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 osds down", 2020-03-11T22:42:26.089 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:42:26.090 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.091 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:42:26.091 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.091 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "osd.1 (root=default,host=smithi142) is down" 2020-03-11T22:42:26.091 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.091 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.092 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:42:26.092 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.092 INFO:tasks.workunit.client.0.smithi142.stdout: "PG_DEGRADED": { 2020-03-11T22:42:26.092 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:26.092 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:42:26.092 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:42:26.092 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 2 2020-03-11T22:42:26.093 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.093 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:42:26.093 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.093 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.0 is stuck undersized for 67s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:26.093 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.093 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.093 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.094 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:42:26.095 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.095 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:42:26.095 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.095 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.095 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:42:26.095 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: "mutes": [] 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_first_committed": 1, 2020-03-11T22:42:26.096 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_last_committed": 1, 2020-03-11T22:42:26.097 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap": { 2020-03-11T22:42:26.097 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:42:26.097 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:42:26.097 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:40:15.368080Z", 2020-03-11T22:42:26.097 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:40:15.368080Z", 2020-03-11T22:42:26.097 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release": 15, 2020-03-11T22:42:26.097 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:42:26.098 INFO:tasks.workunit.client.0.smithi142.stdout: "features": { 2020-03-11T22:42:26.098 INFO:tasks.workunit.client.0.smithi142.stdout: "persistent": [ 2020-03-11T22:42:26.098 INFO:tasks.workunit.client.0.smithi142.stdout: "kraken", 2020-03-11T22:42:26.098 INFO:tasks.workunit.client.0.smithi142.stdout: "luminous", 2020-03-11T22:42:26.098 INFO:tasks.workunit.client.0.smithi142.stdout: "mimic", 2020-03-11T22:42:26.098 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap-prune", 2020-03-11T22:42:26.098 INFO:tasks.workunit.client.0.smithi142.stdout: "nautilus", 2020-03-11T22:42:26.099 INFO:tasks.workunit.client.0.smithi142.stdout: "octopus" 2020-03-11T22:42:26.099 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.099 INFO:tasks.workunit.client.0.smithi142.stdout: "optional": [] 2020-03-11T22:42:26.099 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.099 INFO:tasks.workunit.client.0.smithi142.stdout: "mons": [ 2020-03-11T22:42:26.099 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.099 INFO:tasks.workunit.client.0.smithi142.stdout: "rank": 0, 2020-03-11T22:42:26.100 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "a", 2020-03-11T22:42:26.100 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:42:26.100 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.100 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.100 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.100 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:42:26.100 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 0 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 0, 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 0 2020-03-11T22:42:26.101 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.102 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.102 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.102 INFO:tasks.workunit.client.0.smithi142.stdout: "quorum": [ 2020-03-11T22:42:26.102 INFO:tasks.workunit.client.0.smithi142.stdout: 0 2020-03-11T22:42:26.102 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.102 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap": { 2020-03-11T22:42:26.102 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 40, 2020-03-11T22:42:26.103 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:42:26.103 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:40:16.070548+0000", 2020-03-11T22:42:26.103 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:41:16.556767+0000", 2020-03-11T22:42:26.103 INFO:tasks.workunit.client.0.smithi142.stdout: "last_up_change": "2020-03-11T22:41:14.644622+0000", 2020-03-11T22:42:26.103 INFO:tasks.workunit.client.0.smithi142.stdout: "last_in_change": "2020-03-11T22:41:16.139140+0000", 2020-03-11T22:42:26.103 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:42:26.103 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_num": 5800384, 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_set": [ 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "nodown", 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "noin", 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "noout", 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "pglog_hardlimit", 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snapdirs", 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "recovery_deletes", 2020-03-11T22:42:26.104 INFO:tasks.workunit.client.0.smithi142.stdout: "sortbitwise" 2020-03-11T22:42:26.105 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.105 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_version": 7, 2020-03-11T22:42:26.105 INFO:tasks.workunit.client.0.smithi142.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:42:26.105 INFO:tasks.workunit.client.0.smithi142.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:42:26.105 INFO:tasks.workunit.client.0.smithi142.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:42:26.105 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_snapshot": "", 2020-03-11T22:42:26.105 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_max": 1, 2020-03-11T22:42:26.106 INFO:tasks.workunit.client.0.smithi142.stdout: "max_osd": 3, 2020-03-11T22:42:26.106 INFO:tasks.workunit.client.0.smithi142.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:42:26.106 INFO:tasks.workunit.client.0.smithi142.stdout: "min_compat_client": "jewel", 2020-03-11T22:42:26.106 INFO:tasks.workunit.client.0.smithi142.stdout: "require_osd_release": "octopus", 2020-03-11T22:42:26.106 INFO:tasks.workunit.client.0.smithi142.stdout: "pools": [ 2020-03-11T22:42:26.106 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.106 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:42:26.107 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_name": "test", 2020-03-11T22:42:26.107 INFO:tasks.workunit.client.0.smithi142.stdout: "create_time": "2020-03-11T22:40:37.328602+0000", 2020-03-11T22:42:26.107 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": 1, 2020-03-11T22:42:26.107 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_names": "hashpspool", 2020-03-11T22:42:26.107 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:42:26.107 INFO:tasks.workunit.client.0.smithi142.stdout: "size": 3, 2020-03-11T22:42:26.107 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 2, 2020-03-11T22:42:26.108 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_rule": 0, 2020-03-11T22:42:26.108 INFO:tasks.workunit.client.0.smithi142.stdout: "object_hash": 2, 2020-03-11T22:42:26.108 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:42:26.108 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num": 1, 2020-03-11T22:42:26.108 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num": 1, 2020-03-11T22:42:26.108 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num_target": 1, 2020-03-11T22:42:26.108 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_target": 1, 2020-03-11T22:42:26.109 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_pending": 1, 2020-03-11T22:42:26.109 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pg_merge_meta": { 2020-03-11T22:42:26.109 INFO:tasks.workunit.client.0.smithi142.stdout: "source_pgid": "0.0", 2020-03-11T22:42:26.109 INFO:tasks.workunit.client.0.smithi142.stdout: "ready_epoch": 0, 2020-03-11T22:42:26.109 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_started": 0, 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_clean": 0, 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: "source_version": "0'0", 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: "target_version": "0'0" 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: "last_change": "28", 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend": "0", 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:42:26.110 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:42:26.111 INFO:tasks.workunit.client.0.smithi142.stdout: "auid": 0, 2020-03-11T22:42:26.111 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:42:26.111 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_seq": 0, 2020-03-11T22:42:26.111 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_epoch": 0, 2020-03-11T22:42:26.111 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_snaps": [], 2020-03-11T22:42:26.111 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps": "[]", 2020-03-11T22:42:26.111 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_bytes": 0, 2020-03-11T22:42:26.112 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_objects": 0, 2020-03-11T22:42:26.112 INFO:tasks.workunit.client.0.smithi142.stdout: "tiers": [], 2020-03-11T22:42:26.112 INFO:tasks.workunit.client.0.smithi142.stdout: "tier_of": -1, 2020-03-11T22:42:26.112 INFO:tasks.workunit.client.0.smithi142.stdout: "read_tier": -1, 2020-03-11T22:42:26.112 INFO:tasks.workunit.client.0.smithi142.stdout: "write_tier": -1, 2020-03-11T22:42:26.112 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_mode": "none", 2020-03-11T22:42:26.112 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_bytes": 0, 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_objects": 0, 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_flush_age": 0, 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_evict_age": 0, 2020-03-11T22:42:26.113 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profile": "", 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_params": { 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "none" 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_period": 0, 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_count": 0, 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi142.stdout: "use_gmt_hitset": true, 2020-03-11T22:42:26.114 INFO:tasks.workunit.client.0.smithi142.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:42:26.115 INFO:tasks.workunit.client.0.smithi142.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:42:26.115 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:42:26.115 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:42:26.115 INFO:tasks.workunit.client.0.smithi142.stdout: "grade_table": [], 2020-03-11T22:42:26.115 INFO:tasks.workunit.client.0.smithi142.stdout: "stripe_width": 0, 2020-03-11T22:42:26.115 INFO:tasks.workunit.client.0.smithi142.stdout: "expected_num_objects": 0, 2020-03-11T22:42:26.116 INFO:tasks.workunit.client.0.smithi142.stdout: "fast_read": false, 2020-03-11T22:42:26.116 INFO:tasks.workunit.client.0.smithi142.stdout: "options": {}, 2020-03-11T22:42:26.116 INFO:tasks.workunit.client.0.smithi142.stdout: "application_metadata": {} 2020-03-11T22:42:26.116 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.116 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.116 INFO:tasks.workunit.client.0.smithi142.stdout: "osds": [ 2020-03-11T22:42:26.116 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.117 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:42:26.117 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "99254390-63b0-48e6-be77-d00d55fcfdaa", 2020-03-11T22:42:26.117 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:42:26.117 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:42:26.117 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:42:26.117 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:42:26.117 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 5, 2020-03-11T22:42:26.118 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:42:26.118 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 34, 2020-03-11T22:42:26.118 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 39, 2020-03-11T22:42:26.118 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 29, 2020-03-11T22:42:26.118 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:42:26.118 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.119 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.120 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:42:26.120 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.120 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.120 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.120 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.120 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:42:26.120 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.121 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.121 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.121 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:42:26.121 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.121 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.121 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.121 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.122 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:42:26.122 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.122 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.122 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.122 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.122 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:26.122 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.123 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:42:26.124 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.124 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.124 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.124 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.124 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:26.124 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.124 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.125 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.125 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:42:26.125 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.125 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.125 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.125 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.125 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30274 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6803/30274", 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6805/30274", 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30274", 2020-03-11T22:42:26.126 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30274", 2020-03-11T22:42:26.127 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:42:26.127 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:42:26.127 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:42:26.127 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.127 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.127 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.127 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "277f3e48-aefe-4e39-9f54-e1f3bf3d05ed", 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 0, 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 0, 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 0, 2020-03-11T22:42:26.128 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 10, 2020-03-11T22:42:26.129 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 22, 2020-03-11T22:42:26.129 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 30, 2020-03-11T22:42:26.129 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:42:26.129 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:42:26.129 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.129 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.129 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.130 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:26.130 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.130 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.130 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.130 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.130 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:26.130 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.131 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.131 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.132 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.132 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:42:26.132 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.132 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.132 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.133 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:26.133 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.133 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.133 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.133 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.133 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:26.133 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.134 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.134 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.134 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.134 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:26.134 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.134 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.134 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.135 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:26.135 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.135 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.135 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.135 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.135 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:26.135 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.136 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:26.137 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.137 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.137 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.137 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.137 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:26.137 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 25904 2020-03-11T22:42:26.137 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.138 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.138 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.138 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/25904", 2020-03-11T22:42:26.138 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/25904", 2020-03-11T22:42:26.138 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25904", 2020-03-11T22:42:26.138 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25904", 2020-03-11T22:42:26.138 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: "exists" 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "1ebabf1d-927a-49d6-9158-8e49a3cc24b8", 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:42:26.139 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:42:26.140 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:42:26.140 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:42:26.140 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 14, 2020-03-11T22:42:26.140 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:42:26.140 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 35, 2020-03-11T22:42:26.140 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 0, 2020-03-11T22:42:26.141 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 31, 2020-03-11T22:42:26.141 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:42:26.141 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:42:26.141 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.141 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.141 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.142 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:26.142 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.142 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.142 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.142 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.142 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:26.142 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.143 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.144 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:26.144 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.144 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.144 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.144 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.145 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:26.145 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.145 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.145 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.145 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.145 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:26.146 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.146 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.146 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.146 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:26.146 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.146 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.146 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.147 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.147 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:26.147 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.147 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.147 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.147 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.148 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:26.148 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:42:26.148 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.148 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:42:26.148 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:26.148 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.148 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 30468 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.149 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/30468", 2020-03-11T22:42:26.150 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/30468", 2020-03-11T22:42:26.150 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30468", 2020-03-11T22:42:26.150 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30468", 2020-03-11T22:42:26.150 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:42:26.150 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:42:26.150 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:42:26.150 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.151 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.151 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.151 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_xinfo": [ 2020-03-11T22:42:26.151 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.153 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:42:26.153 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:41:08.565604+0000", 2020-03-11T22:42:26.153 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:42:26.153 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:42:26.153 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:42:26.153 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:42:26.153 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:22.946554+0000", 2020-03-11T22:42:26.154 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:42:26.155 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.155 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.155 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:42:26.155 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:41:09.078441+0000", 2020-03-11T22:42:26.155 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:42:26.155 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:42:26.155 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:29.211602+0000", 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:41:09.599535+0000", 2020-03-11T22:42:26.156 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:42:26.157 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:42:26.157 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:42:26.157 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:42:26.157 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:35.311405+0000", 2020-03-11T22:42:26.157 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:42:26.157 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap": [], 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap_items": [], 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_temp": [], 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_temp": [], 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: "blacklist": {}, 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profiles": { 2020-03-11T22:42:26.158 INFO:tasks.workunit.client.0.smithi142.stdout: "default": { 2020-03-11T22:42:26.159 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:42:26.159 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "2", 2020-03-11T22:42:26.159 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:42:26.159 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van" 2020-03-11T22:42:26.159 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.159 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.159 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps_queue": [], 2020-03-11T22:42:26.160 INFO:tasks.workunit.client.0.smithi142.stdout: "new_removed_snaps": [], 2020-03-11T22:42:26.160 INFO:tasks.workunit.client.0.smithi142.stdout: "new_purged_snaps": [], 2020-03-11T22:42:26.160 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_node_flags": {}, 2020-03-11T22:42:26.161 INFO:tasks.workunit.client.0.smithi142.stdout: "device_class_flags": {} 2020-03-11T22:42:26.161 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.161 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_metadata": [ 2020-03-11T22:42:26.161 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.161 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:42:26.161 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:42:26.161 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6804/30274,v1:127.0.0.1:6805/30274]", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:26.162 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:26.163 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:26.163 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:42:26.163 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:26.163 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:26.163 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:26.163 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:26.163 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:42:26.164 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:26.164 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:26.164 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:26.164 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:42:26.164 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:42:26.164 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:26.164 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:42:26.165 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:42:26.165 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:26.165 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:42:26.166 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6802/30274,v1:127.0.0.1:6803/30274]", 2020-03-11T22:42:26.166 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:42:26.166 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30274,v1:127.0.0.1:6809/30274]", 2020-03-11T22:42:26.166 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30274,v1:127.0.0.1:6807/30274]", 2020-03-11T22:42:26.166 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:42:26.166 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:42:26.166 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:26.167 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:26.167 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:26.167 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:42:26.167 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:26.167 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:26.167 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:42:26.167 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/25904,v1:127.0.0.1:6813/25904]", 2020-03-11T22:42:26.168 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:42:26.169 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:42:26.169 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:26.169 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:26.169 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:26.169 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:26.169 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:26.169 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:26.170 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:26.170 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:42:26.170 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:26.170 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:26.170 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:26.170 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:26.170 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:42:26.171 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:42:26.172 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:26.172 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:42:26.172 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/25904,v1:127.0.0.1:6811/25904]", 2020-03-11T22:42:26.172 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:42:26.172 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25904,v1:127.0.0.1:6817/25904]", 2020-03-11T22:42:26.172 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25904,v1:127.0.0.1:6815/25904]", 2020-03-11T22:42:26.172 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:42:26.173 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:42:26.173 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:26.173 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:26.173 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:26.173 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:42:26.173 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:26.174 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:26.174 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:42:26.174 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:42:26.174 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:26.174 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:42:26.174 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.174 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/30468,v1:127.0.0.1:6813/30468]", 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:26.175 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:26.176 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:26.177 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:26.177 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:42:26.177 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:26.177 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:26.177 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:26.177 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:42:26.177 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/30468,v1:127.0.0.1:6811/30468]", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:42:26.178 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30468,v1:127.0.0.1:6817/30468]", 2020-03-11T22:42:26.179 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30468,v1:127.0.0.1:6815/30468]", 2020-03-11T22:42:26.179 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:42:26.179 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:42:26.179 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:26.179 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:26.179 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:26.179 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.180 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_first_committed": 1, 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_last_committed": 40, 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: "crushmap": { 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": [ 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.0", 2020-03-11T22:42:26.181 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:42:26.182 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.182 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.182 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:42:26.182 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.1", 2020-03-11T22:42:26.182 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:42:26.182 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.182 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.183 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:42:26.183 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.2", 2020-03-11T22:42:26.183 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:42:26.183 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.183 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.183 INFO:tasks.workunit.client.0.smithi142.stdout: "types": [ 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 0, 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd" 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "host" 2020-03-11T22:42:26.184 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.185 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.185 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 2, 2020-03-11T22:42:26.185 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "chassis" 2020-03-11T22:42:26.185 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.185 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.185 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 3, 2020-03-11T22:42:26.185 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "rack" 2020-03-11T22:42:26.186 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.186 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.186 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 4, 2020-03-11T22:42:26.186 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "row" 2020-03-11T22:42:26.186 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.186 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.186 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 5, 2020-03-11T22:42:26.187 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pdu" 2020-03-11T22:42:26.187 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.187 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.187 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 6, 2020-03-11T22:42:26.187 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pod" 2020-03-11T22:42:26.187 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.187 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.188 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 7, 2020-03-11T22:42:26.188 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "room" 2020-03-11T22:42:26.188 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.188 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.188 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 8, 2020-03-11T22:42:26.188 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "datacenter" 2020-03-11T22:42:26.188 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.189 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.189 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 9, 2020-03-11T22:42:26.189 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "zone" 2020-03-11T22:42:26.189 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.189 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.189 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 10, 2020-03-11T22:42:26.190 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "region" 2020-03-11T22:42:26.190 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.190 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.190 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:42:26.190 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "root" 2020-03-11T22:42:26.190 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.190 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.191 INFO:tasks.workunit.client.0.smithi142.stdout: "buckets": [ 2020-03-11T22:42:26.191 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.191 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -1, 2020-03-11T22:42:26.191 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default", 2020-03-11T22:42:26.191 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:42:26.191 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:42:26.191 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:42:26.192 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:42:26.192 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:42:26.192 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:42:26.192 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.192 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:42:26.192 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:42:26.192 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:42:26.193 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.193 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.193 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.193 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.193 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -2, 2020-03-11T22:42:26.194 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default~hdd", 2020-03-11T22:42:26.194 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:42:26.194 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:42:26.194 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:42:26.195 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:42:26.195 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:42:26.195 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:42:26.195 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.195 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:42:26.195 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:42:26.195 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:42:26.196 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.196 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.196 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.196 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.196 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:42:26.196 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142", 2020-03-11T22:42:26.196 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:42:26.197 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:42:26.197 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:42:26.197 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:42:26.197 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:42:26.197 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:42:26.197 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.197 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:42:26.198 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.199 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.199 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:42:26.199 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:42:26.199 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:42:26.199 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.199 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142~hdd", 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:42:26.200 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:42:26.201 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:42:26.201 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:42:26.201 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.201 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:42:26.201 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:42:26.201 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:42:26.201 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.202 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.202 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:42:26.202 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:42:26.202 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:42:26.202 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.202 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.202 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: "rules": [ 2020-03-11T22:42:26.203 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.204 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 0, 2020-03-11T22:42:26.204 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "replicated_rule", 2020-03-11T22:42:26.204 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 0, 2020-03-11T22:42:26.204 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:42:26.204 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 1, 2020-03-11T22:42:26.204 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 10, 2020-03-11T22:42:26.205 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:42:26.205 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.205 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:42:26.205 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:42:26.205 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:42:26.205 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.205 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_firstn", 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.206 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:42:26.207 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.207 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.207 INFO:tasks.workunit.client.0.smithi142.stdout: "tunables": { 2020-03-11T22:42:26.207 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_tries": 0, 2020-03-11T22:42:26.207 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:42:26.207 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_total_tries": 50, 2020-03-11T22:42:26.207 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_stable": 1, 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "straw_calc_version": 1, 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "profile": "jewel", 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "optimal_tunables": 1, 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "legacy_tunables": 0, 2020-03-11T22:42:26.208 INFO:tasks.workunit.client.0.smithi142.stdout: "minimum_required_version": "jewel", 2020-03-11T22:42:26.209 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables": 1, 2020-03-11T22:42:26.209 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables2": 1, 2020-03-11T22:42:26.209 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v2_rules": 0, 2020-03-11T22:42:26.209 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables3": 1, 2020-03-11T22:42:26.209 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v3_rules": 0, 2020-03-11T22:42:26.209 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v4_buckets": 1, 2020-03-11T22:42:26.209 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables5": 1, 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v5_rules": 0 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_args": {} 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: "fsmap": { 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: "default_fscid": -1, 2020-03-11T22:42:26.210 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": { 2020-03-11T22:42:26.211 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": {}, 2020-03-11T22:42:26.211 INFO:tasks.workunit.client.0.smithi142.stdout: "ro_compat": {}, 2020-03-11T22:42:26.211 INFO:tasks.workunit.client.0.smithi142.stdout: "incompat": { 2020-03-11T22:42:26.211 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_1": "base v0.20", 2020-03-11T22:42:26.211 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:42:26.211 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:42:26.211 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_8": "no anchor table", 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_9": "file layout v2", 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.212 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_flags": { 2020-03-11T22:42:26.213 INFO:tasks.workunit.client.0.smithi142.stdout: "enable_multiple": false, 2020-03-11T22:42:26.213 INFO:tasks.workunit.client.0.smithi142.stdout: "ever_enabled_multiple": false 2020-03-11T22:42:26.213 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.213 INFO:tasks.workunit.client.0.smithi142.stdout: "standbys": [], 2020-03-11T22:42:26.213 INFO:tasks.workunit.client.0.smithi142.stdout: "filesystems": [] 2020-03-11T22:42:26.213 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.213 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:42:26.214 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:42:26.214 INFO:tasks.workunit.client.0.smithi142.stdout: "auth": { 2020-03-11T22:42:26.214 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:42:26.214 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 5, 2020-03-11T22:42:26.214 INFO:tasks.workunit.client.0.smithi142.stdout: "num_secrets": 3 2020-03-11T22:42:26.214 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.215 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:42:26.215 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_active": 1, 2020-03-11T22:42:26.215 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_unknown": 0, 2020-03-11T22:42:26.215 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osd": 3, 2020-03-11T22:42:26.215 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_sum": { 2020-03-11T22:42:26.215 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:42:26.215 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 47947, 2020-03-11T22:42:26.216 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:42:26.216 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:42:26.216 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:42:26.216 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:26.216 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:42:26.216 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:26.216 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:26.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:26.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:26.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:42:26.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:42:26.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:42:26.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:42:26.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:42:26.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:26.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:26.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:26.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:26.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:26.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:26.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:26.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:26.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:42:26.221 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 0, 2020-03-11T22:42:26.222 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 0, 2020-03-11T22:42:26.222 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:42:26.222 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:26.222 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:42:26.222 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:42:26.222 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:42:26.222 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 17, 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 17, 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 2, 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 2, 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 0 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum": { 2020-03-11T22:42:26.223 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:42:26.224 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:42:26.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:42:26.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:42:26.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:26.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:26.224 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:42:26.225 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3179328, 2020-03-11T22:42:26.225 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 33280, 2020-03-11T22:42:26.225 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 14, 2020-03-11T22:42:26.225 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145713, 2020-03-11T22:42:26.225 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393472, 2020-03-11T22:42:26.225 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:42:26.225 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:42:26.226 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318866915328, 2020-03-11T22:42:26.226 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:26.226 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 34078720, 2020-03-11T22:42:26.226 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 407796, 2020-03-11T22:42:26.226 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:42:26.226 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:26.226 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 14420, 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221211052 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:26.227 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:26.228 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:42:26.229 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.229 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:42:26.229 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:42:26.229 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.229 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum_by_class": { 2020-03-11T22:42:26.229 INFO:tasks.workunit.client.0.smithi142.stdout: "hdd": { 2020-03-11T22:42:26.229 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3179328, 2020-03-11T22:42:26.230 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 33280, 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 14, 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145713, 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393472, 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318866915328, 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:26.231 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 34078720, 2020-03-11T22:42:26.232 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 407796, 2020-03-11T22:42:26.232 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:42:26.232 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:26.232 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:42:26.232 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 14420, 2020-03-11T22:42:26.232 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221211052 2020-03-11T22:42:26.232 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:42:26.233 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.234 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:42:26.234 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:26.234 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:26.234 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:26.234 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:42:26.234 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.235 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:42:26.235 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:42:26.235 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.235 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.235 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_stats": [ 2020-03-11T22:42:26.235 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.235 INFO:tasks.workunit.client.0.smithi142.stdout: "poolid": 1, 2020-03-11T22:42:26.236 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:42:26.236 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:42:26.236 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 47947, 2020-03-11T22:42:26.236 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:42:26.236 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:42:26.236 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:42:26.236 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:42:26.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:42:26.238 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:42:26.238 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:42:26.238 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:26.238 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:26.238 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:26.238 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:26.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:26.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:26.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:42:26.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:26.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:26.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:42:26.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:42:26.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:42:26.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:42:26.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:42:26.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:26.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:26.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:42:26.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:42:26.242 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.242 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:42:26.242 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:42:26.242 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:42:26.242 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:42:26.242 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 19529728, 2020-03-11T22:42:26.242 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 142522, 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 17, 2020-03-11T22:42:26.243 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 17, 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 2, 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 2, 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 3 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_stats": [ 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.244 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:42:26.245 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 146028888083 2020-03-11T22:42:26.245 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.245 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.245 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:42:26.245 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 42949672975 2020-03-11T22:42:26.246 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.246 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.246 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:42:26.246 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 150323855378 2020-03-11T22:42:26.246 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.246 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.246 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_state": [ 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: "state": "active+undersized+degraded", 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 1 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_osd": [ 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.247 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:42:26.248 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:42:26.248 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:42:26.248 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:26.248 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.248 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.248 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:42:26.248 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 0, 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 1, 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:42:26.249 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": [ 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": {} 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.250 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:42:26.251 INFO:tasks.workunit.client.0.smithi142.stdout: "servicemap": { 2020-03-11T22:42:26.251 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:42:26.251 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:40:16.070967+0000", 2020-03-11T22:42:26.251 INFO:tasks.workunit.client.0.smithi142.stdout: "services": {} 2020-03-11T22:42:26.251 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:42:26.252 INFO:tasks.workunit.client.0.smithi142.stdout: "paxos": { 2020-03-11T22:42:26.252 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:42:26.252 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 377, 2020-03-11T22:42:26.252 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pn": 0, 2020-03-11T22:42:26.252 INFO:tasks.workunit.client.0.smithi142.stdout: "accepted_pn": 0 2020-03-11T22:42:26.252 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:42:26.252 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:42:26.253 INFO:tasks.workunit.client.0.smithi142.stderr:report 2787801924 2020-03-11T22:42:26.254 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:42:26.254 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:42:26.254 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:26.254 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery 2020-03-11T22:42:26.415 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:26.415 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:32.005802+0000 40'101 40:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:42:26.415 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:42:26.415 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:26.416 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:42:26.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:42:26.425 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:42:26.426 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:42:26.426 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:42:26.426 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:26.426 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:26.426 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:42:26.426 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:42:26.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:42:26.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:42:26.517 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:26.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:42:26.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:42:26.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:42:26.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:42:26.525 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:26.525 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:26.526 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:42:26.526 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:42:26.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:42:26.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:42:26.617 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:26.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:42:26.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:26.624 INFO:tasks.workunit.client.0.smithi142.stdout:revive divergent 1 2020-03-11T22:42:26.943 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:26.943 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:32.005802+0000 40'101 40:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:42:26.944 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:42:26.944 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:26.945 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:42:26.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:42:27.512 INFO:tasks.workunit.client.0.smithi142.stderr:noup is set 2020-03-11T22:42:27.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:42:27.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:42:27.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:42:27.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:42:27.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:42:27.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:27.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:27.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:27.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:27.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:27.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:27.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:27.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:27.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:42:27.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:27.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:42:27.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:27.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:27.525 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:27.525 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:27.525 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:42:27.526 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:42:27.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:42:27.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:27.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:27.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:27.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:27.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:27.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:27.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:42:27.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:42:27.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:27.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:42:27.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:27.527 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:42:27.528 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:42:27.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:42:27.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:42:27.529 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:42:27.529 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:42:27.549 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:27.546+0000 7fdb15096ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:27.550 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:27.548+0000 7fdb15096ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:27.552 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:27.550+0000 7fdb15096ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:27.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:42:27.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:42:28.068 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:28.066+0000 7fdb15096ec0 -1 Falling back to public interface 2020-03-11T22:42:28.956 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:28.949+0000 7fdb15096ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:42:32.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:42:32.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:32.888 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery divergent 2020-03-11T22:42:33.217 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:33.217 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:32.005802+0000 40'101 40:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:42:33.217 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:42:33.217 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:33.218 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:42:33.227 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:42:33.228 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:42:33.228 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:42:33.228 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:33.228 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:33.228 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:42:33.228 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:42:33.229 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:42:33.229 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:42:33.318 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:33.325 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:42:34.020 INFO:tasks.workunit.client.0.smithi142.stderr:noup is unset 2020-03-11T22:42:34.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:42:34.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:34.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:34.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:34.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:34.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:34.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:34.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:34.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:34.030 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:42:34.382 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 34 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30274,v1:127.0.0.1:6803/30274] [v2:127.0.0.1:6804/30274,v1:127.0.0.1:6805/30274] exists,up 99254390-63b0-48e6-be77-d00d55fcfdaa 2020-03-11T22:42:34.382 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:42:34.383 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:34.383 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:34.383 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:34.383 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:42:34.383 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:34.383 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:34.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:34.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:34.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:34.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:34.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:34.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:34.723 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33023,v1:127.0.0.1:6819/33023] [v2:127.0.0.1:6820/33023,v1:127.0.0.1:6821/33023] exists,up 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed 2020-03-11T22:42:34.723 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:42:34.723 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:34.723 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:34.723 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:34.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:42:34.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:34.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:34.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:34.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:34.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:34.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:34.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:34.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:35.066 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,28) [v2:127.0.0.1:6810/30468,v1:127.0.0.1:6811/30468] [v2:127.0.0.1:6812/30468,v1:127.0.0.1:6813/30468] exists,up 1ebabf1d-927a-49d6-9158-8e49a3cc24b8 2020-03-11T22:42:35.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:35.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:35.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:35.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:35.392 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:35.393 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:42:33.873702+0000 28'106 44:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:42:35.393 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:42:35.393 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:35.394 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:42:35.402 INFO:tasks.workunit.client.0.smithi142.stdout:wait for peering 2020-03-11T22:42:35.402 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:42:35.403 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:35.722 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:35.722 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2020-03-11T22:42:33.873702+0000 28'106 44:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:42:35.722 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:42:35.723 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:35.724 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:42:35.732 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:42:44.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:42:44.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:44.709 INFO:tasks.workunit.client.0.smithi142.stdout:killing divergent 1 2020-03-11T22:42:45.038 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:45.038 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:42:34.046872+0000 40'101 45:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:42:45.038 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:42:45.038 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:45.039 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:42:45.047 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:42:45.048 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:42:45.048 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:42:45.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:42:45.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:42:45.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:42:45.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:42:45.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17403.out 2020-03-11T22:42:45.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17403.out 2020-03-11T22:42:45.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:42:45.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:42:45.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:42:45.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:42:45.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:42:45.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17403.out 2020-03-11T22:42:45.700 INFO:tasks.workunit.client.0.smithi142.stderr:Exporting 1.0 info 1.0( v 45'102 lc 0'0 (28'90,45'102] local-lis/les=44/45 n=101 ec=22/22 lis/c=44/22 les/c/f=45/23/0 sis=44 pruub=10.636474586s) 2020-03-11T22:42:45.716 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:42:45.734 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:42:45.752 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:42:45.754 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:42:45.756 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:42:45.760 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:42:45.761 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:42:45.762 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:42:45.763 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:42:45.763 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:42:45.764 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:42:45.764 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:42:45.770 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:42:45.771 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:42:45.772 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:42:45.772 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:42:45.774 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:42:45.778 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:42:45.778 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:42:45.779 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:42:45.780 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:42:45.787 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:42:45.791 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:42:45.796 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:42:45.797 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:42:45.797 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:42:45.798 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:42:45.798 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:42:45.799 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:42:45.799 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:42:45.800 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:42:45.803 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:42:45.803 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:42:45.804 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:42:45.804 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:42:45.805 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:42:45.810 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:42:45.811 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:42:45.811 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:42:45.812 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:42:45.813 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:42:45.813 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:42:45.814 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:42:45.814 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:42:45.815 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:42:45.816 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:42:45.816 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:42:45.817 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:42:45.817 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:42:45.818 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:42:45.818 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:42:45.819 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:42:45.819 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:42:45.820 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:42:45.826 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:42:45.826 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:42:45.835 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:42:45.835 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:42:45.836 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:42:45.837 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:42:45.837 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:42:45.838 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:42:45.839 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:42:45.839 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:42:45.840 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:42:45.840 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:42:45.841 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:42:45.842 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:42:45.842 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:42:45.843 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:42:45.843 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:42:45.844 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:42:45.844 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:42:45.845 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:42:45.845 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:42:45.846 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:42:45.846 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:42:45.847 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:42:45.847 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:42:45.848 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:42:45.848 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:42:45.849 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:42:45.858 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:42:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:42:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:42:45.860 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:42:45.862 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:42:45.866 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:42:45.871 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:42:45.871 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:42:45.872 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:42:45.872 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:42:45.872 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:42:45.872 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:42:45.872 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:42:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:42:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:42:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:Export successful 2020-03-11T22:42:45.873 INFO:tasks.workunit.client.0.smithi142.stdout: marking collection for removal 2020-03-11T22:42:45.874 INFO:tasks.workunit.client.0.smithi142.stdout:setting '_remove' omap key 2020-03-11T22:42:45.874 INFO:tasks.workunit.client.0.smithi142.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:42:45.924 INFO:tasks.workunit.client.0.smithi142.stderr:Remove successful 2020-03-11T22:42:46.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17403.out 2020-03-11T22:42:46.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:42:46.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:42:46.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:42:46.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:42:46.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:42:46.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17403.out 2020-03-11T22:42:46.836 INFO:tasks.workunit.client.0.smithi142.stdout:get_pg_num_history pg_num_history pg_num_history(e45 pg_nums {1={22=1}} deleted_pools ) 2020-03-11T22:42:46.836 INFO:tasks.workunit.client.0.smithi142.stdout:Importing pgid 1.0 2020-03-11T22:42:46.836 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:42:46.837 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.890 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:42:46.890 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.898 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:42:46.898 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.906 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:42:46.907 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.915 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:42:46.915 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.935 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:42:46.935 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.951 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:42:46.951 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.968 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:42:46.968 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.985 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:42:46.985 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.990 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:42:46.993 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:46.998 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:42:46.998 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.006 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:42:47.007 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.015 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:42:47.015 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.023 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:42:47.023 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.060 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:42:47.060 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.071 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:42:47.071 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.079 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:42:47.079 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.088 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:42:47.088 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.096 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:42:47.096 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.104 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:42:47.104 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.112 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:42:47.113 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.121 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:42:47.121 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.129 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:42:47.130 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.138 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:42:47.138 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.146 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:42:47.146 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.154 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:42:47.154 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.163 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:42:47.163 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.171 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:42:47.171 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.210 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:42:47.210 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.215 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:42:47.215 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.223 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:42:47.224 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.232 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:42:47.232 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.240 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:42:47.240 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.248 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:42:47.249 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.257 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:42:47.257 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.265 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:42:47.265 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.273 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:42:47.274 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.299 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:42:47.299 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.307 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:42:47.307 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.315 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:42:47.315 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.332 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:42:47.332 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.377 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:42:47.377 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.382 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:42:47.382 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.390 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:42:47.391 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.399 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:42:47.400 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.407 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:42:47.408 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.416 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:42:47.416 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.424 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:42:47.424 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.432 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:42:47.432 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.441 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:42:47.441 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.449 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:42:47.449 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.457 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:42:47.458 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.466 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:42:47.466 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.474 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:42:47.474 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.483 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:42:47.483 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.518 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:42:47.519 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.524 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:42:47.524 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.532 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:42:47.533 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.541 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:42:47.541 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.549 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:42:47.550 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.558 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:42:47.559 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.566 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:42:47.566 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.574 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:42:47.575 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.583 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:42:47.583 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.591 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:42:47.591 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.625 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:42:47.626 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.655 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:42:47.656 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.658 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:42:47.658 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.693 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:42:47.694 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.700 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:42:47.700 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.708 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:42:47.708 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.716 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:42:47.716 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.725 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:42:47.725 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.733 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:42:47.733 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.741 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:42:47.741 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.750 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:42:47.750 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.758 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:42:47.758 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.766 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:42:47.767 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.775 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:42:47.775 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.783 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:42:47.783 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.792 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:42:47.792 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.800 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:42:47.800 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.835 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:42:47.836 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.842 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:42:47.842 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.850 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:42:47.850 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.858 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:42:47.858 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.867 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:42:47.867 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.875 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:42:47.875 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.883 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:42:47.883 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.892 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:42:47.892 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.900 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:42:47.900 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.908 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:42:47.909 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.917 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:42:47.917 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.926 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:42:47.926 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.934 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:42:47.934 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.942 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:42:47.942 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.977 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:42:47.977 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:42:47.983 INFO:tasks.workunit.client.0.smithi142.stdout:write_pg epoch 45 info 1.0( v 45'102 lc 0'0 (28'90,45'102] local-lis/les=44/45 n=101 ec=22/22 lis/c=44/22 les/c/f=45/23/0 sis=44 pruub=10.636474586s) 2020-03-11T22:42:47.992 INFO:tasks.workunit.client.0.smithi142.stdout:Import successful 2020-03-11T22:42:48.515 INFO:tasks.workunit.client.0.smithi142.stdout:reviving divergent 1 2020-03-11T22:42:48.516 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:42:48.516 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:42:48.842 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:48.842 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:42:34.046872+0000 40'101 45:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:42:48.842 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:42:48.842 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:48.843 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:42:48.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:42:48.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:42:48.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:42:48.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:42:48.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:42:48.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:48.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:48.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:48.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:48.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:48.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:48.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:48.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:48.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:42:48.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:48.856 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:42:48.857 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:48.857 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:48.857 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:48.857 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:48.857 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:42:48.857 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:42:48.857 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:42:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:42:48.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:42:48.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:48.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:42:48.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:48.859 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:42:48.859 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:42:48.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:42:48.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:42:48.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:42:48.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:42:48.880 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:48.878+0000 7fca5ee05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:48.882 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:48.879+0000 7fca5ee05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:48.883 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:48.881+0000 7fca5ee05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:49.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:42:49.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:49.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:49.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:49.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:49.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:49.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:49.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:49.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:49.202 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:42:49.398 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:49.395+0000 7fca5ee05ec0 -1 Falling back to public interface 2020-03-11T22:42:49.578 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33023,v1:127.0.0.1:6819/33023] [v2:127.0.0.1:6820/33023,v1:127.0.0.1:6821/33023] exists,up 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed 2020-03-11T22:42:49.578 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:42:49.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:49.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:49.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:49.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:42:49.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:49.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:49.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:49.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:49.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:49.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:49.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:49.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:49.920 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/33023,v1:127.0.0.1:6819/33023] [v2:127.0.0.1:6820/33023,v1:127.0.0.1:6821/33023] exists,up 277f3e48-aefe-4e39-9f54-e1f3bf3d05ed 2020-03-11T22:42:49.920 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:49.920 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:49.920 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:49.921 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:42:50.149 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:42:50.146+0000 7fca5ee05ec0 -1 osd.1 45 log_to_monitors {default=true} 2020-03-11T22:43:09.922 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:43:09.922 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:43:09.922 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:43:09.922 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:09.923 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:09.923 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:09.923 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:43:09.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:43:09.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:43:10.019 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:43:10.019 INFO:tasks.workunit.client.0.smithi142.stdout: "ops": [], 2020-03-11T22:43:10.020 INFO:tasks.workunit.client.0.smithi142.stdout: "num_ops": 0 2020-03-11T22:43:10.020 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:43:10.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:43:10.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:43:10.027 INFO:tasks.workunit.client.0.smithi142.stdout:allowing recovery 2020-03-11T22:43:10.348 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:10.348 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:42:51.493290+0000 45'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:43:10.349 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:43:10.349 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:10.350 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:43:10.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:43:10.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:43:10.479 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:10.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:43:10.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:43:10.606 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:10.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:43:10.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:43:10.734 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:10.742 INFO:tasks.workunit.client.0.smithi142.stdout:reading divergent objects 2020-03-11T22:43:10.742 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:43:10.742 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:43:11.067 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:11.067 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:42:51.493290+0000 45'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:37.378734+0000 0'0 2020-03-11T22:40:37.378734+0000 0 2020-03-11T22:43:11.068 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:43:11.068 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:11.069 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:43:11.078 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:43:11.079 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:43:11.079 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:11.079 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:43:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:43:11.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:11.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:43:11.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:11.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:43:11.191 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:43:11.192 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:11.192 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:43:11.219 INFO:tasks.workunit.client.0.smithi142.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:43:11.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:11.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:43:11.247 INFO:tasks.workunit.client.0.smithi142.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:43:11.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:43:11.249 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:11.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:11.253 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:40:57.850501+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:11.253 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4539.0:1 2020-03-11T22:40:57.850501+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:11.253 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:11.254 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2020-03-11T22:43:11.254 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: more recent entry found: 40'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4650.0:1 2020-03-11T22:41:51.417506+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:43:11.254 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4526.0:1 2020-03-11T22:40:57.829964+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.254 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: keeping 28'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4526.0:1 2020-03-11T22:40:57.829964+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.254 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:11.255 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:43:11.255 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:43:11.255 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:43:11.255 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:40:57.848813+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:11.255 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: keeping 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:40:57.848813+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:11.255 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:43:11.255 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'105 last_divergent_update: 28'105 2020-03-11T22:43:11.256 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:43:11.256 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:11.256 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2020-03-11T22:43:11.256 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4538.0:1 2020-03-11T22:40:57.848813+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:11.256 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:43:11.256 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2020-03-11T22:43:11.256 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'103 (28'2) modify 1:97db2a0f:::existing_2:head by client.4534.0:1 2020-03-11T22:40:57.845650+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.257 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: keeping 28'103 (28'2) modify 1:97db2a0f:::existing_2:head by client.4534.0:1 2020-03-11T22:40:57.845650+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.257 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:11.257 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:43:11.257 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:43:11.257 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:11.257 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2020-03-11T22:43:11.257 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'103 (28'2) modify 1:97db2a0f:::existing_2:head by client.4534.0:1 2020-03-11T22:40:57.845650+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.258 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:43:11.258 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2020-03-11T22:43:11.258 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'104 (28'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:40:57.846566+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.258 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: keeping 28'104 (28'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:40:57.846566+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.258 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:11.259 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:43:11.259 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:43:11.259 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:11.259 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2020-03-11T22:43:11.260 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'104 (28'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:40:57.846566+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.260 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:43:11.260 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2020-03-11T22:43:11.260 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4527.0:1 2020-03-11T22:40:57.835031+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.260 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 20 _merge_object_divergent_entries: keeping 28'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4527.0:1 2020-03-11T22:40:57.835031+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:11.260 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:11.260 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'102 last_divergent_update: 28'102 2020-03-11T22:43:11.261 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:43:11.261 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:34.022+0000 7fdaf3251700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:43:11.261 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:11.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:11.261 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:11.262 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:11.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:43:11.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17403.out 2020-03-11T22:43:11.263 INFO:tasks.workunit.client.0.smithi142.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17403.out': No such file or directory 2020-03-11T22:43:11.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:43:11.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:43:11.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:43:11.264 INFO:tasks.workunit.client.0.smithi142.stdout:success 2020-03-11T22:43:11.726 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' does not exist 2020-03-11T22:43:11.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:43:11.735 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:11.735 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:11.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:11.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:11.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:11.842 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:11.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:43:11.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:43:11.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:43:11.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:43:11.843 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:11.843 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:11.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:11.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:11.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:11.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:11.848 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:43:11.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:43:11.849 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:43:11.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:43:11.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:43:11.850 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:43:11.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:11.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:43:11.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:43:11.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:11.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:43:11.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:43:11.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:43:11.865 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:43:11.865 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:11.866 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:11.866 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:43:11.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:43:11.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:43:11.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:43:11.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:43:11.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:43:11.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:43:11.869 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:43:11.869 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:43:11.869 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:43:11.869 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:11.869 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:11.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:11.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:11.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:11.871 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:11.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:43:11.871 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:43:11.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:43:11.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:43:11.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:43:11.872 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:43:11.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:11.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:43:11.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:43:11.873 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:11.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:43:11.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:43:11.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:43:11.876 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:43:11.876 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:11.876 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:11.876 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:43:11.877 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:43:11.877 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:43:11.877 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:43:11.878 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:43:11.878 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:11.878 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:11.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17403 2020-03-11T22:43:11.880 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:43:11.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:43:11.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:43:11.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:43:11.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:43:11.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:43:11.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:43:11.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:43:11.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:43:11.882 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:43:11.882 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:43:11.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:43:11.883 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:43:11.883 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:43:11.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:43:11.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:43:11.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:43:11.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:43:11.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:43:11.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:43:11.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:43:12.149 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:43:12.149 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:12.149 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:12.149 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:12.150 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:12.150 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:12.150 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:12.150 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:43:12.328 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:43:12.329 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:43:12.329 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:43:12.329 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:43:12.329 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:43:12.330 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:43:12.331 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:43:12.331 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:43:12.331 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:43:12.331 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:12.331 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:12.332 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:12.332 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:43:12.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:43:12.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get fsid 2020-03-11T22:43:12.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:43:12.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:43:12.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:43:12.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:43:12.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:43:12.419 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:43:12.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:43:12.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:43:12.420 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:12.420 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:12.420 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:12.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:43:12.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:43:12.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get mon_host 2020-03-11T22:43:12.504 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:43:12.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:43:12.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:43:12.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:43:12.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:43:12.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:43:12.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:43:13.383 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:43:13.384 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:13.384 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:13.384 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:13.385 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:13.385 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:13.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:13.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:43:13.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:43:13.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:43:13.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:43:13.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:13.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:13.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:43:13.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:13.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:13.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:13.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:13.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:13.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:13.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:13.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:13.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:13.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:13.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:13.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:13.415 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:13.415 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:13.415 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:13.415 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:13.415 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:13.415 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:13.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:13.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:13.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:13.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:13.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:13.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:13.418 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:13.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=65f689f5-c5c4-499f-8f76-b5d7d974b0d8 2020-03-11T22:43:13.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 65f689f5-c5c4-499f-8f76-b5d7d974b0d8' 2020-03-11T22:43:13.419 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:13.420 INFO:tasks.workunit.client.0.smithi142.stdout:add osd0 65f689f5-c5c4-499f-8f76-b5d7d974b0d8 2020-03-11T22:43:13.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCBaWlewr+LGRAAKY+QRpEDSEgOQFFlHPNQbA== 2020-03-11T22:43:13.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCBaWlewr+LGRAAKY+QRpEDSEgOQFFlHPNQbA=="}' 2020-03-11T22:43:13.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 65f689f5-c5c4-499f-8f76-b5d7d974b0d8 -i td/divergent-priors/0/new.json 2020-03-11T22:43:13.782 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:43:13.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:43:13.793 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCBaWlewr+LGRAAKY+QRpEDSEgOQFFlHPNQbA== --osd-uuid 65f689f5-c5c4-499f-8f76-b5d7d974b0d8 2020-03-11T22:43:13.821 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:13.818+0000 7fc74971bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:13.823 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:13.820+0000 7fc74971bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:13.824 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:13.821+0000 7fc74971bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:13.825 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:13.822+0000 7fc74971bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:13.825 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:13.822+0000 7fc74971bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:13.825 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:13.822+0000 7fc74971bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:13.866 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:13.863+0000 7fc74971bec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:43:16.801 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:43:16.801 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:16.802 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd0 key to auth repository 2020-03-11T22:43:16.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:43:16.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:17.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:43:17.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:17.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:17.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:17.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:17.249 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:43:17.274 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:17.271+0000 7f35c0c99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:17.276 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:17.273+0000 7f35c0c99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:17.277 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:17.274+0000 7f35c0c99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:17.600 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:43:17.600 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:17.600 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:17.600 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:17.600 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:17.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:17.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:17.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:17.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:17.601 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:43:17.792 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:17.789+0000 7f35c0c99ec0 -1 Falling back to public interface 2020-03-11T22:43:17.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:18.541 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:18.538+0000 7f35c0c99ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:43:18.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:18.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:18.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:18.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:18.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:18.955 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:43:19.303 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34518,v1:127.0.0.1:6803/34518] [v2:127.0.0.1:6804/34518,v1:127.0.0.1:6805/34518] exists,up 65f689f5-c5c4-499f-8f76-b5d7d974b0d8 2020-03-11T22:43:19.303 INFO:tasks.workunit.client.0.smithi142.stdout:add osd1 e79d27b4-6fc2-43ed-90f7-84fe096e508b 2020-03-11T22:43:19.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:19.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:19.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:19.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:43:19.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:43:19.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:19.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:19.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:43:19.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:19.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:43:19.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:19.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:19.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:19.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:19.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:43:19.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:43:19.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:19.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:19.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:19.307 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:19.307 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:19.307 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:19.307 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:19.308 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:19.308 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:19.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:19.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:19.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:19.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:19.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:19.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:19.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:19.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:19.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:19.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:19.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:19.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:19.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:43:19.310 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:19.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e79d27b4-6fc2-43ed-90f7-84fe096e508b 2020-03-11T22:43:19.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 e79d27b4-6fc2-43ed-90f7-84fe096e508b' 2020-03-11T22:43:19.310 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:19.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCHaWleG20CExAAncGgz6OekioHU0Ualbankw== 2020-03-11T22:43:19.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCHaWleG20CExAAncGgz6OekioHU0Ualbankw=="}' 2020-03-11T22:43:19.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e79d27b4-6fc2-43ed-90f7-84fe096e508b -i td/divergent-priors/1/new.json 2020-03-11T22:43:19.785 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:43:19.795 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:43:19.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCHaWleG20CExAAncGgz6OekioHU0Ualbankw== --osd-uuid e79d27b4-6fc2-43ed-90f7-84fe096e508b 2020-03-11T22:43:19.825 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:19.822+0000 7f726c906ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:19.826 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:19.823+0000 7f726c906ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:19.827 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:19.825+0000 7f726c906ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:19.828 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:19.825+0000 7f726c906ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:19.828 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:19.825+0000 7f726c906ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:19.828 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:19.825+0000 7f726c906ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:19.913 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:19.910+0000 7f726c906ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:43:22.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:43:22.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:22.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:43:22.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:22.886 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd1 key to auth repository 2020-03-11T22:43:23.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:43:23.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:23.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:23.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:23.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:23.344 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:43:23.369 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:23.365+0000 7fdb052abec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:23.371 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:23.367+0000 7fdb052abec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:23.372 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:23.369+0000 7fdb052abec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:23.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:43:23.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:23.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:23.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:23.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:23.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:23.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:23.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:23.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:23.728 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:43:23.888 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:23.884+0000 7fdb052abec0 -1 Falling back to public interface 2020-03-11T22:43:24.073 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:24.597 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:24.593+0000 7fdb052abec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:43:25.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:25.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:25.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:25.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:25.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:25.076 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:43:25.435 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34873,v1:127.0.0.1:6811/34873] [v2:127.0.0.1:6812/34873,v1:127.0.0.1:6813/34873] exists,up e79d27b4-6fc2-43ed-90f7-84fe096e508b 2020-03-11T22:43:25.435 INFO:tasks.workunit.client.0.smithi142.stdout:add osd2 19594912-e0ba-4ab5-9da4-a2f9818068ee 2020-03-11T22:43:25.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:25.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:25.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:25.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:43:25.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:43:25.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:25.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:25.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:43:25.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:25.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:25.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:25.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:25.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:25.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:25.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:25.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:25.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:25.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:25.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:25.439 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:25.439 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:25.439 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:25.440 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:25.440 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:25.440 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:25.440 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:25.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:43:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:25.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:25.443 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:25.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=19594912-e0ba-4ab5-9da4-a2f9818068ee 2020-03-11T22:43:25.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 19594912-e0ba-4ab5-9da4-a2f9818068ee' 2020-03-11T22:43:25.443 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:25.454 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCNaWlexVXHGhAADpKQF572VEcjsjSUKAAtdw== 2020-03-11T22:43:25.455 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCNaWlexVXHGhAADpKQF572VEcjsjSUKAAtdw=="}' 2020-03-11T22:43:25.455 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 19594912-e0ba-4ab5-9da4-a2f9818068ee -i td/divergent-priors/2/new.json 2020-03-11T22:43:25.900 INFO:tasks.workunit.client.0.smithi142.stdout:2 2020-03-11T22:43:25.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:43:25.911 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCNaWlexVXHGhAADpKQF572VEcjsjSUKAAtdw== --osd-uuid 19594912-e0ba-4ab5-9da4-a2f9818068ee 2020-03-11T22:43:25.939 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:25.935+0000 7fe4b4491ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:25.971 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:25.967+0000 7fe4b4491ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:25.972 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:25.968+0000 7fe4b4491ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:25.973 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:25.969+0000 7fe4b4491ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:25.973 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:25.969+0000 7fe4b4491ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:25.973 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:25.969+0000 7fe4b4491ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:26.029 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:26.026+0000 7fe4b4491ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:43:28.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:43:28.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:28.976 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd2 key to auth repository 2020-03-11T22:43:28.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:43:28.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:29.433 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:43:29.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:43:29.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:29.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:29.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:29.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:29.459 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:29.455+0000 7fe1da56cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:29.461 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:29.457+0000 7fe1da56cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:29.462 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:29.458+0000 7fe1da56cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:29.779 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:43:29.779 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:29.779 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:29.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:29.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:29.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:29.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:29.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:29.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:29.781 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:43:29.979 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:29.976+0000 7fe1da56cec0 -1 Falling back to public interface 2020-03-11T22:43:30.145 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:30.721 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:43:30.717+0000 7fe1da56cec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:43:31.148 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:31.149 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:31.149 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:31.149 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:31.149 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:31.149 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:43:31.496 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35228,v1:127.0.0.1:6819/35228] [v2:127.0.0.1:6820/35228,v1:127.0.0.1:6821/35228] exists,up 19594912-e0ba-4ab5-9da4-a2f9818068ee 2020-03-11T22:43:31.496 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:31.496 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:31.496 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:31.496 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:43:32.188 INFO:tasks.workunit.client.0.smithi142.stderr:noout is set 2020-03-11T22:43:32.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:43:32.882 INFO:tasks.workunit.client.0.smithi142.stderr:noin is set 2020-03-11T22:43:32.890 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:43:33.558 INFO:tasks.workunit.client.0.smithi142.stderr:nodown is set 2020-03-11T22:43:33.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:43:33.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:43:34.037 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' already exists 2020-03-11T22:43:34.044 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:43:35.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:43:35.695 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 size to 3 2020-03-11T22:43:35.703 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:43:36.330 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 min_size to 2 2020-03-11T22:43:36.338 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:43:37.040 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:43:37.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:43:37.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:37.049 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:37.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:37.393 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:43:37.393 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:43:37.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:37.394 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:37.394 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:37.530 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:43:37.530 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:43:37.530 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:37.530 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:37.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-03-11T22:43:37.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2020-03-11T22:43:37.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:37.659 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:37.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-03-11T22:43:37.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-60129542148' 2020-03-11T22:43:37.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:37.788 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:43:37.788 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:37.788 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:37.789 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:43:37.789 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:37.790 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:43:37.790 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:43:37.790 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:37.790 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:43:38.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-11T22:43:38.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:39.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:39.134 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:39.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:43:39.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:39.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705669 2020-03-11T22:43:39.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:39.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:39.480 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705669 2020-03-11T22:43:39.480 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:39.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705669 2020-03-11T22:43:39.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-03-11T22:43:39.481 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 38654705669 2020-03-11T22:43:39.482 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:39.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705669 -lt 38654705669 2020-03-11T22:43:39.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:39.826 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542148 2020-03-11T22:43:39.826 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:39.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:39.827 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542148 2020-03-11T22:43:39.827 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:39.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542148 2020-03-11T22:43:39.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-03-11T22:43:39.829 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 60129542148 2020-03-11T22:43:39.829 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:40.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542148 2020-03-11T22:43:40.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:43:40.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:40.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:40.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:40.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:40.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:43:40.172 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:40.172 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:40.173 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:40.173 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:40.173 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:40.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:40.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:40.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:40.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:40.264 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:40.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:40.610 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:43:40.610 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:43:40.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:40.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:40.611 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:40.785 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:43:40.785 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:43:40.786 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:40.786 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:40.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705671 2020-03-11T22:43:40.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705671' 2020-03-11T22:43:40.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:40.914 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:41.043 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-11T22:43:41.043 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705671 2-60129542150' 2020-03-11T22:43:41.043 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:41.044 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:43:41.044 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:41.044 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:41.045 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:43:41.045 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:41.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:43:41.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:43:41.046 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:41.046 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:43:41.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:43:41.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:42.392 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:42.393 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:42.773 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:43:42.773 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:43.773 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:43:43.774 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:44.121 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:43:44.121 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:44.122 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705671 2020-03-11T22:43:44.122 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:44.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:44.123 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705671 2020-03-11T22:43:44.123 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:44.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705671 2020-03-11T22:43:44.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2020-03-11T22:43:44.124 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:44.125 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 38654705671 2020-03-11T22:43:44.468 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705672 -lt 38654705671 2020-03-11T22:43:44.468 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:44.469 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-11T22:43:44.469 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:44.469 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:44.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-11T22:43:44.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:44.474 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-11T22:43:44.474 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-11T22:43:44.474 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:44.474 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 60129542150 2020-03-11T22:43:44.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542151 -lt 60129542150 2020-03-11T22:43:44.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:44.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:44.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:45.262 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:45.262 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:45.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:45.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:45.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:45.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:45.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:45.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:45.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:43:45.592 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:45.592 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:45.592 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:46.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:43:46.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:43:46.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:43:46.030 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:43:46.031 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:43:46.350 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:43:46.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:43:46.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:43:46.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:46.360 INFO:tasks.workunit.client.0.smithi142.stdout:primary and soon to be divergent is 1 2020-03-11T22:43:46.679 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:46.679 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:43:34.894453+0000 0'0 28:14 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:43:46.679 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:43:46.679 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:46.681 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:43:46.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:43:46.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:43:46.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:43:46.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:43:46.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:43:46.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:43:46.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:43:46.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:43:46.691 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:43:46.691 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:43:46.691 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:43:46.691 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:43:46.691 INFO:tasks.workunit.client.0.smithi142.stdout:writing initial objects 2020-03-11T22:43:46.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:46.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:43:46.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:46.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:43:46.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:46.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:43:46.846 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:46.846 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:43:46.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:46.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:43:46.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:46.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:43:47.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:43:47.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:43:47.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:43:47.138 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.138 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:43:47.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:43:47.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:43:47.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:43:47.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:43:47.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:43:47.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:43:47.521 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:43:47.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:43:47.664 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.664 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:43:47.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:43:47.755 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.755 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:43:47.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:43:47.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:43:47.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:47.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:43:48.014 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.015 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:43:48.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:43:48.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:43:48.147 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.147 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:43:48.188 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.188 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:43:48.230 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.230 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:43:48.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:43:48.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:43:48.430 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.430 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:43:48.471 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.472 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:43:48.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:43:48.691 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.691 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:43:48.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:43:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:43:48.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:43:48.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:43:48.989 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:48.989 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:43:49.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:43:49.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:43:49.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:43:49.205 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.205 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:43:49.283 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.284 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:43:49.330 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.330 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:43:49.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:43:49.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:43:49.530 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.531 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:43:49.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:43:49.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.614 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:43:49.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:43:49.742 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:43:49.789 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.789 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:43:49.830 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.831 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:43:49.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:43:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:43:50.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:43:50.114 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.114 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:43:50.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.156 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:43:50.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:43:50.239 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.239 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:43:50.280 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:43:50.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:43:50.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:43:50.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:43:50.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:43:50.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:43:50.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:43:50.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:43:50.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.798 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:43:50.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:43:50.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:43:50.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:43:50.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:50.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:43:51.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:43:51.089 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.089 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:43:51.131 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.131 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:43:51.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:43:51.254 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.254 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:43:51.298 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.298 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:43:51.378 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.379 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:43:51.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:43:51.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:43:51.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:43:51.548 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.548 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:43:51.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:43:51.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:43:51.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:43:51.906 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.906 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:43:51.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:51.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:43:52.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:43:52.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:43:52.148 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.148 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:43:52.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:43:52.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:43:52.273 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.273 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:43:52.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:43:52.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:43:52.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:43:52.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:43:52.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:43:52.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:52.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:52.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:52.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:52.441 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:43:52.442 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:52.442 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:52.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:52.443 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:52.443 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:52.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:52.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:52.498 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:52.498 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:52.498 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:52.842 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:43:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:43:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:52.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:52.843 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:52.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:43:52.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:43:52.997 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:52.997 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:53.125 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-11T22:43:53.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-38654705676' 2020-03-11T22:43:53.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:53.126 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:53.254 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-03-11T22:43:53.255 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-38654705676 2-60129542155' 2020-03-11T22:43:53.255 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:53.255 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:43:53.255 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:53.256 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:53.256 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:43:53.256 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:53.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:43:53.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:43:53.258 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:53.258 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:43:53.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836494 2020-03-11T22:43:53.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:54.603 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:54.603 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:54.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-11T22:43:54.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:54.956 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-11T22:43:54.956 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:54.957 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:54.957 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-11T22:43:54.957 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:54.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-11T22:43:54.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-11T22:43:54.958 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:54.959 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 38654705676 2020-03-11T22:43:55.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705676 -lt 38654705676 2020-03-11T22:43:55.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:55.305 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542155 2020-03-11T22:43:55.305 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:55.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:55.307 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542155 2020-03-11T22:43:55.307 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:55.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542155 2020-03-11T22:43:55.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-03-11T22:43:55.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:55.309 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 60129542155 2020-03-11T22:43:55.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542155 2020-03-11T22:43:55.647 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:55.647 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:55.648 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:56.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:56.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:56.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:56.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:56.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:56.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:56.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:56.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:56.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:43:56.425 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:56.425 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:56.425 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:56.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:43:56.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:43:56.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:43:56.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:43:56.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:43:56.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:43:56.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:43:56.872 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:43:57.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:43:57.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:43:57.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:57.220 INFO:tasks.workunit.client.0.smithi142.stdout:blackholing osds 0 2 2020-03-11T22:43:57.563 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:57.564 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:43:34.894453+0000 30'100 30:114 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:43:57.564 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:43:57.564 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:57.565 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:43:57.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:57.576 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:43:57.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:43:57.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:43:57.577 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:57.577 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:57.577 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:57.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:43:57.577 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:43:57.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:43:57.661 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:43:57.661 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:57.661 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:43:57.668 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:43:57.669 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:43:57.669 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:43:57.669 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:43:57.669 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:57.669 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:57.670 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:43:57.670 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:43:57.670 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:43:57.670 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:43:57.755 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:43:57.756 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:43:57.756 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:43:57.763 INFO:tasks.workunit.client.0.smithi142.stdout:create new divergent objects 2020-03-11T22:43:57.763 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:43:57.763 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:43:57.764 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:57.764 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:57.764 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:43:57.764 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:43:57.765 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:43:57.765 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:43:57.765 INFO:tasks.workunit.client.0.smithi142.stdout:writing divergent objects 2020-03-11T22:43:57.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:43:57.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:43:57.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:43:57.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:43:57.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:43:57.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:43:57.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:43:57.767 INFO:tasks.workunit.client.0.smithi142.stdout:remove divergent objects 2020-03-11T22:43:57.768 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:43:57.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:43:57.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:43:57.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:43:57.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:43:57.770 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:44:07.771 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:44:07.776 INFO:tasks.workunit.client.0.smithi142.stderr:rados: no process found 2020-03-11T22:44:07.776 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:44:07.777 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:07.777 INFO:tasks.workunit.client.0.smithi142.stdout:killing all the osds 2020-03-11T22:44:08.107 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:08.107 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:43:34.894453+0000 30'106 30:120 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:08.108 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:44:08.108 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:08.109 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:44:08.117 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:44:08.118 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:08.118 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:08.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:08.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:08.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:08.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:08.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:44:08.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:44:08.720 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already down. 2020-03-11T22:44:08.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:44:08.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:44:09.254 INFO:tasks.workunit.client.0.smithi142.stderr:osd.1 is already down. 2020-03-11T22:44:09.262 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:44:09.262 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:44:09.770 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already down. 2020-03-11T22:44:09.778 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:09.778 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:44:10.263 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already out. 2020-03-11T22:44:10.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:10.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:44:10.764 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already out. 2020-03-11T22:44:10.772 INFO:tasks.workunit.client.0.smithi142.stdout:bringing up non_divergent 0 2 2020-03-11T22:44:10.772 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:44:10.772 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:11.091 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:11.092 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:43:34.894453+0000 30'106 30:120 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:11.092 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:44:11.092 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:11.093 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:44:11.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:11.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:44:11.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:11.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:11.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:44:11.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:11.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:44:11.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:11.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:11.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:11.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:11.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:44:11.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:44:11.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:11.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:11.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:11.107 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:11.107 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:11.107 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:11.107 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:11.107 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:11.107 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:44:11.107 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:44:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:44:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:11.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:11.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:11.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:11.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:44:11.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:44:11.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:11.110 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:44:11.110 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:44:11.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:44:11.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:11.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:11.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:11.131 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:11.128+0000 7f444aee1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:11.132 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:11.130+0000 7f444aee1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:11.133 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:11.130+0000 7f444aee1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:11.451 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:44:11.451 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:11.451 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:11.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:11.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:11.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:11.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:11.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:11.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:11.452 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:44:11.648 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:11.646+0000 7f444aee1ec0 -1 Falling back to public interface 2020-03-11T22:44:11.794 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:12.472 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:12.470+0000 7f444aee1ec0 -1 osd.0 30 log_to_monitors {default=true} 2020-03-11T22:44:12.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:12.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:12.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:12.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:12.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:12.797 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:44:13.162 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39340,v1:127.0.0.1:6803/39340] [v2:127.0.0.1:6804/39340,v1:127.0.0.1:6805/39340] exists,up 65f689f5-c5c4-499f-8f76-b5d7d974b0d8 2020-03-11T22:44:13.162 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:44:13.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:13.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:13.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:13.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:13.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:44:13.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:13.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:13.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:44:13.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:13.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:44:13.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:13.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:13.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:13.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:13.166 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:44:13.167 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:44:13.167 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:13.167 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:13.167 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:13.167 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:13.168 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:13.168 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:13.168 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:13.168 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:13.169 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:44:13.169 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:44:13.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:44:13.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:13.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:13.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:13.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:13.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:13.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:13.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:13.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:13.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:44:13.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:44:13.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:13.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:44:13.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:44:13.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:13.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:13.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:13.185 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:13.182+0000 7f8db16c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:13.187 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:13.184+0000 7f8db16c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:13.189 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:13.186+0000 7f8db16c2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:13.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:44:13.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:13.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:13.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:13.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:13.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:13.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:13.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:13.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:13.584 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:44:13.707 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:13.704+0000 7f8db16c2ec0 -1 Falling back to public interface 2020-03-11T22:44:13.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:14.576 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:44:14.573+0000 7f8db16c2ec0 -1 osd.2 30 log_to_monitors {default=true} 2020-03-11T22:44:14.925 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:14.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:14.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:14.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:14.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:14.926 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:44:15.268 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 33 last_clean_interval [14,30) [v2:127.0.0.1:6810/39534,v1:127.0.0.1:6811/39534] [v2:127.0.0.1:6812/39534,v1:127.0.0.1:6813/39534] exists,up 19594912-e0ba-4ab5-9da4-a2f9818068ee 2020-03-11T22:44:15.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:15.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:15.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:15.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:15.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:44:16.059 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already in. 2020-03-11T22:44:16.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:44:16.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:44:16.614 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already in. 2020-03-11T22:44:16.623 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:16.623 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:44:16.623 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:16.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:16.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:16.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:16.624 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:16.624 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:16.625 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:16.625 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:16.625 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:16.625 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:16.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:16.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:16.680 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:16.680 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:16.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:17.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:17.029 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:44:17.029 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:44:17.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:17.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:17.030 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:17.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:44:17.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659' 2020-03-11T22:44:17.224 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:17.225 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:17.248 INFO:tasks.workunit.client.0.smithi142.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:44:17.256 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:44:17.256 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1-' 2020-03-11T22:44:17.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:17.257 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:17.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789955 2020-03-11T22:44:17.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822659 1- 2-158913789955' 2020-03-11T22:44:17.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:17.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822659 2020-03-11T22:44:17.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:17.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:17.387 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822659 2020-03-11T22:44:17.387 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:17.388 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:44:17.388 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2020-03-11T22:44:17.389 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:17.389 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 154618822659 2020-03-11T22:44:17.755 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 154618822659 2020-03-11T22:44:17.755 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:18.756 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:18.757 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:19.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822660 -lt 154618822659 2020-03-11T22:44:19.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:19.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:44:19.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:19.129 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:19.129 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:44:19.129 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:19.129 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:44:19.130 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:44:19.130 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:19.130 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 2020-03-11T22:44:19.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 2020-03-11T22:44:19.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705679: unary operator expected 2020-03-11T22:44:19.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:19.481 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789955 2020-03-11T22:44:19.481 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:19.482 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:19.482 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789955 2020-03-11T22:44:19.483 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:19.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789955 2020-03-11T22:44:19.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2020-03-11T22:44:19.484 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 158913789955 2020-03-11T22:44:19.485 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:19.824 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789955 -lt 158913789955 2020-03-11T22:44:19.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:19.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:19.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:20.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:20.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:20.271 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:20.271 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:20.271 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:20.271 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:20.272 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:20.272 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:20.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:20.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:20.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:20.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:21.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:21.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:44:21.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:44:21.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:44:21.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:21.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:44:21.146 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:21.146 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:21.147 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:21.147 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:21.148 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:21.148 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:21.148 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:21.148 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:21.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:21.480 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:21.481 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:21.481 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:21.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:21.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:21.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:21.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:21.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:21.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:21.925 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:21.925 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:21.925 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:22.375 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:22.375 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:22.375 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:22.375 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:22.376 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:22.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:22.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:22.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:22.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:22.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:22.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:22.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:22.577 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:22.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:22.914 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:22.915 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:22.915 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:23.357 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:23.357 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:23.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:23.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:23.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:23.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:23.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:23.358 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:23.358 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:23.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:23.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:23.798 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:23.798 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:23.798 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:24.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:24.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:24.199 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:24.199 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:24.200 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:24.200 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:24.200 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:24.200 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:24.537 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:24.538 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:24.538 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:24.538 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:24.983 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:24.983 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:24.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:24.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:24.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:24.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:24.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:24.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:24.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:26.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:26.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:26.243 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:26.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:26.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:26.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:26.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:26.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:26.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:26.592 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:26.592 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:26.592 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:27.037 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:27.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:27.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:27.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:27.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:27.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:27.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:27.038 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:27.039 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:27.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:27.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:27.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:44:29.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:29.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:29.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:29.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:29.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:29.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:29.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:29.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:29.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:29.432 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:29.432 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:29.432 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:29.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:29.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:29.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:29.876 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:29.876 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:29.876 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:29.876 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:29.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:29.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:30.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:44:33.521 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:33.521 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:33.522 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:33.522 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:33.522 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:33.522 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:33.522 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:33.523 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:33.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:33.863 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:33.864 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:33.864 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:34.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:34.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:34.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:34.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:34.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:34.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:34.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:34.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:34.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:34.764 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:34.765 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:34.765 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:44:34.765 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:34.765 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:44:41.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:41.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:41.166 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:41.167 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:41.167 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:41.167 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:41.167 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:41.168 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:41.510 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:41.510 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:41.511 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:41.511 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:41.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:41.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:41.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:41.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:41.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:41.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:41.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:41.955 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:41.955 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:42.395 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:42.395 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:42.395 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:44:42.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:42.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:44:49.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:49.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:49.697 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:49.697 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:49.698 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:49.698 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:49.698 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:49.698 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:50.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:50.040 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:50.040 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:50.040 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:50.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:50.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:50.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:50.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:50.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:50.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:50.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:50.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:50.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:50.921 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:50.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:50.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:44:50.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:44:51.359 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:44:51.359 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_fingerprint": "e56ab1ef-8a74-41c4-a177-661e9fcb9197", 2020-03-11T22:44:51.360 INFO:tasks.workunit.client.0.smithi142.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:44:51.360 INFO:tasks.workunit.client.0.smithi142.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:44:51.360 INFO:tasks.workunit.client.0.smithi142.stdout: "timestamp": "2020-03-11T22:44:51.356025+0000", 2020-03-11T22:44:51.360 INFO:tasks.workunit.client.0.smithi142.stdout: "tag": "", 2020-03-11T22:44:51.360 INFO:tasks.workunit.client.0.smithi142.stdout: "health": { 2020-03-11T22:44:51.360 INFO:tasks.workunit.client.0.smithi142.stdout: "status": "HEALTH_WARN", 2020-03-11T22:44:51.361 INFO:tasks.workunit.client.0.smithi142.stdout: "checks": { 2020-03-11T22:44:51.361 INFO:tasks.workunit.client.0.smithi142.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:44:51.361 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:51.361 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:44:51.361 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:44:51.362 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 17 2020-03-11T22:44:51.362 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.362 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [], 2020-03-11T22:44:51.362 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:44:51.362 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.363 INFO:tasks.workunit.client.0.smithi142.stdout: "OSD_DOWN": { 2020-03-11T22:44:51.363 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:51.363 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:44:51.363 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 osds down", 2020-03-11T22:44:51.364 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:44:51.364 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.364 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:44:51.364 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.365 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "osd.1 (root=default,host=smithi142) is down" 2020-03-11T22:44:51.365 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.365 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.365 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:44:51.366 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.366 INFO:tasks.workunit.client.0.smithi142.stdout: "PG_DEGRADED": { 2020-03-11T22:44:51.366 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:51.366 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:44:51.367 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:44:51.367 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 32 2020-03-11T22:44:51.367 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.367 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:44:51.367 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.368 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.368 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.368 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.368 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.369 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.369 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.369 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.369 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.370 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.370 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.370 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.370 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.371 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.371 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.371 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.371 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.372 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.372 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.372 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.372 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.372 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.373 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.373 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.373 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.373 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.374 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.374 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.374 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.374 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.375 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.375 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.375 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.375 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.376 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.376 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.376 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.376 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.377 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.377 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.377 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.377 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.377 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.378 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.378 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.378 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.378 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.379 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.379 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.379 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.379 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.380 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.380 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.380 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.380 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.381 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.381 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.381 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.381 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.382 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.382 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.382 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.382 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.382 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.383 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.383 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.383 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.383 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.384 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.384 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.384 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.385 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.385 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.385 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.385 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.386 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.386 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.386 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.386 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.387 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.387 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.387 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.387 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.388 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.388 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.388 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.388 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.388 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.389 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.389 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.389 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.390 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.390 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.390 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.390 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:44:51.391 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.391 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.391 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:44:51.391 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.392 INFO:tasks.workunit.client.0.smithi142.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:44:51.392 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:44:51.392 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:44:51.392 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:44:51.393 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:44:51.393 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.393 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:44:51.393 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.394 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:44:51.394 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.394 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.394 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:44:51.395 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.395 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.395 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:44:51.395 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.395 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.396 INFO:tasks.workunit.client.0.smithi142.stdout: "mutes": [] 2020-03-11T22:44:51.396 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.396 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_first_committed": 1, 2020-03-11T22:44:51.396 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_last_committed": 1, 2020-03-11T22:44:51.397 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap": { 2020-03-11T22:44:51.397 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:44:51.397 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:44:51.397 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:43:11.901399Z", 2020-03-11T22:44:51.398 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:43:11.901399Z", 2020-03-11T22:44:51.398 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release": 15, 2020-03-11T22:44:51.398 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:44:51.398 INFO:tasks.workunit.client.0.smithi142.stdout: "features": { 2020-03-11T22:44:51.399 INFO:tasks.workunit.client.0.smithi142.stdout: "persistent": [ 2020-03-11T22:44:51.399 INFO:tasks.workunit.client.0.smithi142.stdout: "kraken", 2020-03-11T22:44:51.399 INFO:tasks.workunit.client.0.smithi142.stdout: "luminous", 2020-03-11T22:44:51.399 INFO:tasks.workunit.client.0.smithi142.stdout: "mimic", 2020-03-11T22:44:51.399 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap-prune", 2020-03-11T22:44:51.400 INFO:tasks.workunit.client.0.smithi142.stdout: "nautilus", 2020-03-11T22:44:51.400 INFO:tasks.workunit.client.0.smithi142.stdout: "octopus" 2020-03-11T22:44:51.400 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.400 INFO:tasks.workunit.client.0.smithi142.stdout: "optional": [] 2020-03-11T22:44:51.401 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.401 INFO:tasks.workunit.client.0.smithi142.stdout: "mons": [ 2020-03-11T22:44:51.401 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.402 INFO:tasks.workunit.client.0.smithi142.stdout: "rank": 0, 2020-03-11T22:44:51.402 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "a", 2020-03-11T22:44:51.402 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:44:51.402 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.403 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.403 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.403 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:44:51.403 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 0 2020-03-11T22:44:51.403 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.404 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.404 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.404 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:44:51.404 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:44:51.405 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 0, 2020-03-11T22:44:51.405 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 0 2020-03-11T22:44:51.405 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.405 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.406 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.406 INFO:tasks.workunit.client.0.smithi142.stdout: "quorum": [ 2020-03-11T22:44:51.406 INFO:tasks.workunit.client.0.smithi142.stdout: 0 2020-03-11T22:44:51.406 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.406 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap": { 2020-03-11T22:44:51.407 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 43, 2020-03-11T22:44:51.407 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:44:51.407 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:43:12.586713+0000", 2020-03-11T22:44:51.407 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:44:18.977612+0000", 2020-03-11T22:44:51.408 INFO:tasks.workunit.client.0.smithi142.stdout: "last_up_change": "2020-03-11T22:44:14.647540+0000", 2020-03-11T22:44:51.408 INFO:tasks.workunit.client.0.smithi142.stdout: "last_in_change": "2020-03-11T22:44:16.361638+0000", 2020-03-11T22:44:51.408 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:44:51.408 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_num": 5800384, 2020-03-11T22:44:51.409 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_set": [ 2020-03-11T22:44:51.409 INFO:tasks.workunit.client.0.smithi142.stdout: "nodown", 2020-03-11T22:44:51.409 INFO:tasks.workunit.client.0.smithi142.stdout: "noin", 2020-03-11T22:44:51.409 INFO:tasks.workunit.client.0.smithi142.stdout: "noout", 2020-03-11T22:44:51.410 INFO:tasks.workunit.client.0.smithi142.stdout: "pglog_hardlimit", 2020-03-11T22:44:51.410 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snapdirs", 2020-03-11T22:44:51.410 INFO:tasks.workunit.client.0.smithi142.stdout: "recovery_deletes", 2020-03-11T22:44:51.410 INFO:tasks.workunit.client.0.smithi142.stdout: "sortbitwise" 2020-03-11T22:44:51.411 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.411 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_version": 7, 2020-03-11T22:44:51.411 INFO:tasks.workunit.client.0.smithi142.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:44:51.411 INFO:tasks.workunit.client.0.smithi142.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:44:51.412 INFO:tasks.workunit.client.0.smithi142.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:44:51.412 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_snapshot": "", 2020-03-11T22:44:51.412 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_max": 1, 2020-03-11T22:44:51.412 INFO:tasks.workunit.client.0.smithi142.stdout: "max_osd": 3, 2020-03-11T22:44:51.412 INFO:tasks.workunit.client.0.smithi142.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:44:51.413 INFO:tasks.workunit.client.0.smithi142.stdout: "min_compat_client": "jewel", 2020-03-11T22:44:51.413 INFO:tasks.workunit.client.0.smithi142.stdout: "require_osd_release": "octopus", 2020-03-11T22:44:51.413 INFO:tasks.workunit.client.0.smithi142.stdout: "pools": [ 2020-03-11T22:44:51.414 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.414 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:44:51.414 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_name": "test", 2020-03-11T22:44:51.414 INFO:tasks.workunit.client.0.smithi142.stdout: "create_time": "2020-03-11T22:43:33.821631+0000", 2020-03-11T22:44:51.414 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": 1, 2020-03-11T22:44:51.415 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_names": "hashpspool", 2020-03-11T22:44:51.415 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:44:51.415 INFO:tasks.workunit.client.0.smithi142.stdout: "size": 3, 2020-03-11T22:44:51.415 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 2, 2020-03-11T22:44:51.416 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_rule": 0, 2020-03-11T22:44:51.416 INFO:tasks.workunit.client.0.smithi142.stdout: "object_hash": 2, 2020-03-11T22:44:51.416 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:44:51.416 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num": 32, 2020-03-11T22:44:51.417 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num": 1, 2020-03-11T22:44:51.417 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num_target": 32, 2020-03-11T22:44:51.417 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_target": 32, 2020-03-11T22:44:51.417 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_pending": 32, 2020-03-11T22:44:51.418 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pg_merge_meta": { 2020-03-11T22:44:51.418 INFO:tasks.workunit.client.0.smithi142.stdout: "source_pgid": "0.0", 2020-03-11T22:44:51.418 INFO:tasks.workunit.client.0.smithi142.stdout: "ready_epoch": 0, 2020-03-11T22:44:51.418 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_started": 0, 2020-03-11T22:44:51.419 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_clean": 0, 2020-03-11T22:44:51.419 INFO:tasks.workunit.client.0.smithi142.stdout: "source_version": "0'0", 2020-03-11T22:44:51.419 INFO:tasks.workunit.client.0.smithi142.stdout: "target_version": "0'0" 2020-03-11T22:44:51.419 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.419 INFO:tasks.workunit.client.0.smithi142.stdout: "last_change": "42", 2020-03-11T22:44:51.420 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend": "0", 2020-03-11T22:44:51.420 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:44:51.420 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_preluminous": "42", 2020-03-11T22:44:51.420 INFO:tasks.workunit.client.0.smithi142.stdout: "auid": 0, 2020-03-11T22:44:51.421 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:44:51.421 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_seq": 0, 2020-03-11T22:44:51.421 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_epoch": 0, 2020-03-11T22:44:51.421 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_snaps": [], 2020-03-11T22:44:51.422 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps": "[]", 2020-03-11T22:44:51.422 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_bytes": 0, 2020-03-11T22:44:51.422 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_objects": 0, 2020-03-11T22:44:51.422 INFO:tasks.workunit.client.0.smithi142.stdout: "tiers": [], 2020-03-11T22:44:51.423 INFO:tasks.workunit.client.0.smithi142.stdout: "tier_of": -1, 2020-03-11T22:44:51.423 INFO:tasks.workunit.client.0.smithi142.stdout: "read_tier": -1, 2020-03-11T22:44:51.423 INFO:tasks.workunit.client.0.smithi142.stdout: "write_tier": -1, 2020-03-11T22:44:51.423 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_mode": "none", 2020-03-11T22:44:51.423 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_bytes": 0, 2020-03-11T22:44:51.424 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_objects": 0, 2020-03-11T22:44:51.424 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:44:51.424 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:44:51.424 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:44:51.425 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_flush_age": 0, 2020-03-11T22:44:51.425 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_evict_age": 0, 2020-03-11T22:44:51.425 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profile": "", 2020-03-11T22:44:51.425 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_params": { 2020-03-11T22:44:51.426 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "none" 2020-03-11T22:44:51.426 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.426 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_period": 0, 2020-03-11T22:44:51.426 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_count": 0, 2020-03-11T22:44:51.427 INFO:tasks.workunit.client.0.smithi142.stdout: "use_gmt_hitset": true, 2020-03-11T22:44:51.427 INFO:tasks.workunit.client.0.smithi142.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:44:51.427 INFO:tasks.workunit.client.0.smithi142.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:44:51.427 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:44:51.428 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:44:51.428 INFO:tasks.workunit.client.0.smithi142.stdout: "grade_table": [], 2020-03-11T22:44:51.428 INFO:tasks.workunit.client.0.smithi142.stdout: "stripe_width": 0, 2020-03-11T22:44:51.428 INFO:tasks.workunit.client.0.smithi142.stdout: "expected_num_objects": 0, 2020-03-11T22:44:51.429 INFO:tasks.workunit.client.0.smithi142.stdout: "fast_read": false, 2020-03-11T22:44:51.429 INFO:tasks.workunit.client.0.smithi142.stdout: "options": {}, 2020-03-11T22:44:51.429 INFO:tasks.workunit.client.0.smithi142.stdout: "application_metadata": {} 2020-03-11T22:44:51.429 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.429 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.430 INFO:tasks.workunit.client.0.smithi142.stdout: "osds": [ 2020-03-11T22:44:51.430 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.430 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:44:51.430 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "65f689f5-c5c4-499f-8f76-b5d7d974b0d8", 2020-03-11T22:44:51.431 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:44:51.431 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:44:51.431 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:44:51.431 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:44:51.432 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 5, 2020-03-11T22:44:51.432 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 30, 2020-03-11T22:44:51.432 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 36, 2020-03-11T22:44:51.432 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 42, 2020-03-11T22:44:51.432 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 31, 2020-03-11T22:44:51.433 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:44:51.433 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:44:51.433 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.433 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.434 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.434 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:44:51.434 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.434 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.434 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.434 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.435 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.436 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.436 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:44:51.436 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.436 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.436 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.436 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.436 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:44:51.437 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.437 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.437 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.437 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.437 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:51.437 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.437 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.438 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.438 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:44:51.438 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.438 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.438 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.438 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.438 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:44:51.439 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.439 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.439 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.439 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.439 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:51.439 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.439 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:44:51.440 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:44:51.441 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.441 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.441 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.441 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6803/39340", 2020-03-11T22:44:51.441 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6805/39340", 2020-03-11T22:44:51.441 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39340", 2020-03-11T22:44:51.441 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39340", 2020-03-11T22:44:51.442 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:44:51.442 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:44:51.442 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:44:51.442 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.442 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.442 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.442 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "e79d27b4-6fc2-43ed-90f7-84fe096e508b", 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 0, 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 0, 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 0, 2020-03-11T22:44:51.443 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 9, 2020-03-11T22:44:51.444 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 22, 2020-03-11T22:44:51.444 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 32, 2020-03-11T22:44:51.444 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:44:51.444 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:44:51.444 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.444 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.444 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.445 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:51.445 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.445 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.445 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.445 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.445 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:51.445 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.446 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.446 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.446 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.446 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:44:51.446 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.446 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.447 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.447 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:51.447 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.447 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.447 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.447 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.447 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.448 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.449 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:51.449 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.449 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.449 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.449 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.449 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:51.449 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.450 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.450 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.450 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.450 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:51.450 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.450 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.450 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.451 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:51.451 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.451 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.451 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.451 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.451 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:51.452 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:44:51.452 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.452 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.452 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.452 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/34873", 2020-03-11T22:44:51.452 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/34873", 2020-03-11T22:44:51.452 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34873", 2020-03-11T22:44:51.453 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34873", 2020-03-11T22:44:51.453 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:44:51.453 INFO:tasks.workunit.client.0.smithi142.stdout: "exists" 2020-03-11T22:44:51.453 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.453 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.453 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.453 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:44:51.454 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "19594912-e0ba-4ab5-9da4-a2f9818068ee", 2020-03-11T22:44:51.454 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:44:51.454 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:44:51.454 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:44:51.454 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:44:51.454 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 14, 2020-03-11T22:44:51.454 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 30, 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 37, 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 0, 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 33, 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.455 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.456 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:44:51.456 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.456 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.456 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.456 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.456 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:44:51.456 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.457 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.457 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.457 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.457 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:44:51.457 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.457 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.457 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.458 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.459 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.459 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.459 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:44:51.459 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.459 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.459 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.459 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:44:51.460 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.460 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.460 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.460 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.460 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:44:51.460 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.460 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.461 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.461 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.461 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:44:51.461 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:44:51.461 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.461 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:44:51.461 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:44:51.462 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.462 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.462 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.464 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:44:51.464 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:44:51.464 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:44:51.464 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.464 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.465 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.465 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/39534", 2020-03-11T22:44:51.465 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/39534", 2020-03-11T22:44:51.465 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39534", 2020-03-11T22:44:51.466 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39534", 2020-03-11T22:44:51.466 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:44:51.467 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:44:51.467 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:44:51.467 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.467 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.467 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.467 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_xinfo": [ 2020-03-11T22:44:51.467 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.468 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:44:51.468 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:44:08.519711+0000", 2020-03-11T22:44:51.468 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:44:51.468 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:44:51.468 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:44:51.468 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:44:51.468 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:19.534845+0000", 2020-03-11T22:44:51.469 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:44:51.469 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.469 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.469 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:44:51.469 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:44:09.043221+0000", 2020-03-11T22:44:51.469 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:44:51.469 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:44:51.470 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:44:51.470 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:44:51.470 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:25.625940+0000", 2020-03-11T22:44:51.470 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:44:51.470 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.470 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.470 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:44:51.471 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:44:09.557846+0000", 2020-03-11T22:44:51.471 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:44:51.471 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:44:51.471 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:44:51.471 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:44:51.471 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:31.707590+0000", 2020-03-11T22:44:51.471 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:44:51.472 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.472 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.472 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap": [], 2020-03-11T22:44:51.472 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap_items": [], 2020-03-11T22:44:51.472 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_temp": [], 2020-03-11T22:44:51.472 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_temp": [], 2020-03-11T22:44:51.472 INFO:tasks.workunit.client.0.smithi142.stdout: "blacklist": {}, 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profiles": { 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: "default": { 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "2", 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van" 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.473 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.474 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps_queue": [], 2020-03-11T22:44:51.474 INFO:tasks.workunit.client.0.smithi142.stdout: "new_removed_snaps": [], 2020-03-11T22:44:51.474 INFO:tasks.workunit.client.0.smithi142.stdout: "new_purged_snaps": [], 2020-03-11T22:44:51.474 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_node_flags": {}, 2020-03-11T22:44:51.474 INFO:tasks.workunit.client.0.smithi142.stdout: "device_class_flags": {} 2020-03-11T22:44:51.474 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.474 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_metadata": [ 2020-03-11T22:44:51.475 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.475 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:44:51.475 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:44:51.475 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6804/39340,v1:127.0.0.1:6805/39340]", 2020-03-11T22:44:51.475 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:44:51.475 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:44:51.475 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:44:51.476 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:51.477 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:51.477 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:51.477 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:51.477 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:44:51.477 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:51.477 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:51.477 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:51.478 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:44:51.478 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:44:51.478 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:51.478 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:44:51.478 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:44:51.478 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:51.478 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:44:51.479 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6802/39340,v1:127.0.0.1:6803/39340]", 2020-03-11T22:44:51.479 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:44:51.479 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39340,v1:127.0.0.1:6809/39340]", 2020-03-11T22:44:51.479 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39340,v1:127.0.0.1:6807/39340]", 2020-03-11T22:44:51.479 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:44:51.479 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:44:51.479 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:51.480 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:51.480 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:51.480 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:44:51.480 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:51.480 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:51.480 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:44:51.480 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/34873,v1:127.0.0.1:6813/34873]", 2020-03-11T22:44:51.481 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:44:51.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:44:51.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:51.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:51.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:51.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:51.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:51.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:51.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:51.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:44:51.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:51.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:51.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:51.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:51.483 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:44:51.484 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:51.484 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:51.484 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:51.484 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:44:51.484 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:44:51.484 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:51.485 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:44:51.485 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:44:51.485 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:51.485 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:44:51.485 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/34873,v1:127.0.0.1:6811/34873]", 2020-03-11T22:44:51.485 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:44:51.485 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34873,v1:127.0.0.1:6817/34873]", 2020-03-11T22:44:51.486 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34873,v1:127.0.0.1:6815/34873]", 2020-03-11T22:44:51.486 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:44:51.486 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:44:51.486 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:51.486 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:51.486 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:51.486 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:44:51.487 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:51.487 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:51.487 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:44:51.487 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:44:51.487 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:51.487 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:44:51.487 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.488 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.488 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:44:51.488 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:44:51.488 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/39534,v1:127.0.0.1:6813/39534]", 2020-03-11T22:44:51.488 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:44:51.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:44:51.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:44:51.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:44:51.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:44:51.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:44:51.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:44:51.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:44:51.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:44:51.490 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:44:51.490 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:44:51.490 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:44:51.490 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:44:51.490 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:44:51.490 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:44:51.490 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:44:51.491 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:44:51.491 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:44:51.491 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:44:51.491 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:44:51.491 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:44:51.491 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:44:51.491 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/39534,v1:127.0.0.1:6811/39534]", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39534,v1:127.0.0.1:6817/39534]", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39534,v1:127.0.0.1:6815/39534]", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:44:51.492 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:44:51.493 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:44:51.493 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:44:51.493 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:44:51.493 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:44:51.493 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:44:51.493 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:44:51.493 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:44:51.494 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:44:51.494 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:44:51.494 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:44:51.494 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.494 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.494 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_first_committed": 1, 2020-03-11T22:44:51.494 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_last_committed": 43, 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: "crushmap": { 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": [ 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.0", 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.495 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.496 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:44:51.496 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.1", 2020-03-11T22:44:51.496 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:44:51.496 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.496 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.496 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:44:51.496 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.2", 2020-03-11T22:44:51.497 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:44:51.497 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.497 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.497 INFO:tasks.workunit.client.0.smithi142.stdout: "types": [ 2020-03-11T22:44:51.497 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.497 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 0, 2020-03-11T22:44:51.497 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd" 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "host" 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 2, 2020-03-11T22:44:51.498 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "chassis" 2020-03-11T22:44:51.499 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.499 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.499 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 3, 2020-03-11T22:44:51.499 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "rack" 2020-03-11T22:44:51.499 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.499 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.499 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 4, 2020-03-11T22:44:51.500 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "row" 2020-03-11T22:44:51.500 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.500 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.500 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 5, 2020-03-11T22:44:51.500 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pdu" 2020-03-11T22:44:51.500 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.500 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.501 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 6, 2020-03-11T22:44:51.501 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pod" 2020-03-11T22:44:51.501 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.501 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.501 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 7, 2020-03-11T22:44:51.501 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "room" 2020-03-11T22:44:51.501 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.502 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.502 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 8, 2020-03-11T22:44:51.502 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "datacenter" 2020-03-11T22:44:51.502 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.502 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.502 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 9, 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "zone" 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 10, 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "region" 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.503 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:44:51.504 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "root" 2020-03-11T22:44:51.504 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.504 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.504 INFO:tasks.workunit.client.0.smithi142.stdout: "buckets": [ 2020-03-11T22:44:51.504 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.504 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -1, 2020-03-11T22:44:51.504 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default", 2020-03-11T22:44:51.505 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:44:51.505 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:44:51.505 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:44:51.505 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:44:51.505 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:44:51.505 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:44:51.505 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.506 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -2, 2020-03-11T22:44:51.507 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default~hdd", 2020-03-11T22:44:51.507 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:44:51.507 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:44:51.507 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:44:51.507 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:44:51.507 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:44:51.507 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:44:51.508 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.508 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:44:51.508 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:44:51.508 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:44:51.508 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.508 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.508 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142", 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:44:51.509 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:44:51.510 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:44:51.510 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.510 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:44:51.510 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:44:51.510 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:44:51.510 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.510 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.511 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:44:51.511 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:44:51.511 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:44:51.511 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.511 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.511 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:44:51.511 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142~hdd", 2020-03-11T22:44:51.512 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:44:51.513 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:44:51.513 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:44:51.513 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:44:51.513 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:44:51.513 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:44:51.513 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.513 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:44:51.514 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:44:51.514 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:44:51.514 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.514 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.514 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:44:51.514 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:44:51.514 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.515 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.516 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.516 INFO:tasks.workunit.client.0.smithi142.stdout: "rules": [ 2020-03-11T22:44:51.516 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.516 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 0, 2020-03-11T22:44:51.516 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "replicated_rule", 2020-03-11T22:44:51.516 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 0, 2020-03-11T22:44:51.516 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:44:51.517 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 1, 2020-03-11T22:44:51.517 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 10, 2020-03-11T22:44:51.517 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:44:51.517 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.517 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:44:51.517 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:44:51.517 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_firstn", 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.518 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:44:51.519 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.519 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:44:51.519 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.519 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.519 INFO:tasks.workunit.client.0.smithi142.stdout: "tunables": { 2020-03-11T22:44:51.519 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_tries": 0, 2020-03-11T22:44:51.519 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:44:51.520 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_total_tries": 50, 2020-03-11T22:44:51.520 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:44:51.520 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:44:51.520 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_stable": 1, 2020-03-11T22:44:51.520 INFO:tasks.workunit.client.0.smithi142.stdout: "straw_calc_version": 1, 2020-03-11T22:44:51.520 INFO:tasks.workunit.client.0.smithi142.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:44:51.520 INFO:tasks.workunit.client.0.smithi142.stdout: "profile": "jewel", 2020-03-11T22:44:51.521 INFO:tasks.workunit.client.0.smithi142.stdout: "optimal_tunables": 1, 2020-03-11T22:44:51.521 INFO:tasks.workunit.client.0.smithi142.stdout: "legacy_tunables": 0, 2020-03-11T22:44:51.521 INFO:tasks.workunit.client.0.smithi142.stdout: "minimum_required_version": "jewel", 2020-03-11T22:44:51.521 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables": 1, 2020-03-11T22:44:51.521 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables2": 1, 2020-03-11T22:44:51.521 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v2_rules": 0, 2020-03-11T22:44:51.521 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables3": 1, 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v3_rules": 0, 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v4_buckets": 1, 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables5": 1, 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v5_rules": 0 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_args": {} 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.522 INFO:tasks.workunit.client.0.smithi142.stdout: "fsmap": { 2020-03-11T22:44:51.523 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:44:51.523 INFO:tasks.workunit.client.0.smithi142.stdout: "default_fscid": -1, 2020-03-11T22:44:51.523 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": { 2020-03-11T22:44:51.523 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": {}, 2020-03-11T22:44:51.523 INFO:tasks.workunit.client.0.smithi142.stdout: "ro_compat": {}, 2020-03-11T22:44:51.523 INFO:tasks.workunit.client.0.smithi142.stdout: "incompat": { 2020-03-11T22:44:51.523 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_1": "base v0.20", 2020-03-11T22:44:51.524 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:44:51.524 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:44:51.524 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:44:51.524 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:44:51.524 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:44:51.524 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_8": "no anchor table", 2020-03-11T22:44:51.524 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_9": "file layout v2", 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_flags": { 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: "enable_multiple": false, 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: "ever_enabled_multiple": false 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.525 INFO:tasks.workunit.client.0.smithi142.stdout: "standbys": [], 2020-03-11T22:44:51.526 INFO:tasks.workunit.client.0.smithi142.stdout: "filesystems": [] 2020-03-11T22:44:51.526 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.526 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:44:51.526 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:44:51.526 INFO:tasks.workunit.client.0.smithi142.stdout: "auth": { 2020-03-11T22:44:51.526 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:44:51.526 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 5, 2020-03-11T22:44:51.527 INFO:tasks.workunit.client.0.smithi142.stdout: "num_secrets": 3 2020-03-11T22:44:51.527 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.527 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 32, 2020-03-11T22:44:51.527 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_active": 32, 2020-03-11T22:44:51.527 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_unknown": 0, 2020-03-11T22:44:51.527 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osd": 3, 2020-03-11T22:44:51.527 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_sum": { 2020-03-11T22:44:51.528 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:44:51.528 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48300, 2020-03-11T22:44:51.528 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:44:51.528 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:44:51.528 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:44:51.528 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:51.528 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:44:51.529 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 100, 2020-03-11T22:44:51.530 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 100, 2020-03-11T22:44:51.530 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:51.530 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:51.530 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:51.530 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:51.530 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:51.530 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:51.531 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:44:51.531 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:51.531 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:51.531 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:44:51.531 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:44:51.531 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:44:51.531 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:51.532 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:51.533 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:51.533 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:51.533 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:44:51.533 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:44:51.533 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.533 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:44:51.533 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:44:51.534 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:44:51.534 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:44:51.534 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 0, 2020-03-11T22:44:51.534 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 0, 2020-03-11T22:44:51.534 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:44:51.534 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:51.534 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 512, 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 512, 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 64, 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 64, 2020-03-11T22:44:51.535 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 0 2020-03-11T22:44:51.536 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.536 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum": { 2020-03-11T22:44:51.536 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:44:51.536 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:44:51.536 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 65, 2020-03-11T22:44:51.536 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:44:51.536 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:51.537 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:51.537 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:44:51.537 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3180288, 2020-03-11T22:44:51.537 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 34240, 2020-03-11T22:44:51.537 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 6, 2020-03-11T22:44:51.537 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145721, 2020-03-11T22:44:51.537 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311392512, 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318865932288, 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 35061760, 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 434166, 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:44:51.538 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 6916, 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221218556 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:51.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:44:51.540 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.541 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum_by_class": { 2020-03-11T22:44:51.542 INFO:tasks.workunit.client.0.smithi142.stdout: "hdd": { 2020-03-11T22:44:51.542 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:44:51.542 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:44:51.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 65, 2020-03-11T22:44:51.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:44:51.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:44:51.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:44:51.543 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:44:51.543 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3180288, 2020-03-11T22:44:51.543 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 34240, 2020-03-11T22:44:51.543 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 6, 2020-03-11T22:44:51.543 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145721, 2020-03-11T22:44:51.543 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311392512, 2020-03-11T22:44:51.543 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318865932288, 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 35061760, 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 434166, 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:51.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:44:51.545 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 6916, 2020-03-11T22:44:51.545 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221218556 2020-03-11T22:44:51.545 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.545 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:44:51.545 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:44:51.545 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:44:51.545 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:44:51.546 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:44:51.546 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:44:51.546 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:44:51.546 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.546 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:44:51.546 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:44:51.547 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:44:51.547 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:44:51.547 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:44:51.547 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.547 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:44:51.547 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:44:51.547 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.548 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.548 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_stats": [ 2020-03-11T22:44:51.548 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.548 INFO:tasks.workunit.client.0.smithi142.stdout: "poolid": 1, 2020-03-11T22:44:51.548 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 32, 2020-03-11T22:44:51.548 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:44:51.548 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48300, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:44:51.549 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:44:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:44:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:44:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:44:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:44:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 100, 2020-03-11T22:44:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 100, 2020-03-11T22:44:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:44:51.551 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:44:51.551 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:44:51.551 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:44:51.551 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:44:51.551 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:44:51.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:44:51.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:44:51.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:44:51.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:44:51.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:44:51.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:44:51.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:44:51.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:44:51.554 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:44:51.554 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:44:51.554 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:44:51.554 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:44:51.554 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.554 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:44:51.554 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:44:51.555 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:44:51.555 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:44:51.555 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 19529728, 2020-03-11T22:44:51.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 143228, 2020-03-11T22:44:51.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:44:51.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:44:51.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 512, 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 512, 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 64, 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 64, 2020-03-11T22:44:51.556 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 3 2020-03-11T22:44:51.557 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.557 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.557 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_stats": [ 2020-03-11T22:44:51.557 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.557 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:44:51.557 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 154618822666 2020-03-11T22:44:51.557 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.558 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.558 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:44:51.558 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 38654705679 2020-03-11T22:44:51.558 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.558 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.558 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:44:51.558 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 158913789962 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_state": [ 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: "state": "active+undersized+degraded", 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 32 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.559 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_osd": [ 2020-03-11T22:44:51.560 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.560 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:44:51.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:44:51.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 32, 2020-03-11T22:44:51.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:51.560 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.561 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.561 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:44:51.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:44:51.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 0, 2020-03-11T22:44:51.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:51.561 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.561 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 32, 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 32, 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": [ 2020-03-11T22:44:51.562 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:44:51.563 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:44:51.563 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": {} 2020-03-11T22:44:51.564 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.564 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:44:51.565 INFO:tasks.workunit.client.0.smithi142.stdout: "servicemap": { 2020-03-11T22:44:51.565 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:44:51.565 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:43:12.587022+0000", 2020-03-11T22:44:51.565 INFO:tasks.workunit.client.0.smithi142.stdout: "services": {} 2020-03-11T22:44:51.565 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:44:51.565 INFO:tasks.workunit.client.0.smithi142.stdout: "paxos": { 2020-03-11T22:44:51.565 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:44:51.566 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 316, 2020-03-11T22:44:51.566 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pn": 0, 2020-03-11T22:44:51.566 INFO:tasks.workunit.client.0.smithi142.stdout: "accepted_pn": 0 2020-03-11T22:44:51.566 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:44:51.566 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:44:51.567 INFO:tasks.workunit.client.0.smithi142.stderr:report 2042437357 2020-03-11T22:44:51.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:44:51.567 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:44:51.568 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:44:51.568 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:44:51.568 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:44:51.568 INFO:tasks.workunit.client.0.smithi142.stdout:writing non-divergent object existing_4 2020-03-11T22:44:51.689 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:51.689 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282996+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.689 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281033+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.689 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281891+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.690 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282998+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.690 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281529+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.690 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279177+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.690 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279169+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.690 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281526+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.690 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282095+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.690 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281173+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.691 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280241+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.691 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279827+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.691 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281136+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.691 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281718+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.691 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280838+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.691 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280668+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.692 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280205+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.692 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280456+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.692 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279651+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.692 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280405+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.692 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:32.641933+0000 30'100 43:134 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.692 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281745+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.693 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282114+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.693 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279429+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.693 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279199+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.693 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279163+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.693 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281429+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.693 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279830+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.693 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280031+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.694 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280292+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.694 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279863+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.694 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280584+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:44:51.694 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:44:51.694 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:51.694 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:44:51.698 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:44:51.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:51.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:44:51.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:51.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:51.745 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:51.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:51.746 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:51.747 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:51.747 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:51.747 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:51.747 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:51.747 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:51.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:51.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:51.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:51.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:51.803 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:52.144 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:52.145 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:44:52.145 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:44:52.145 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:52.145 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:52.145 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:52.275 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822668 2020-03-11T22:44:52.275 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822668' 2020-03-11T22:44:52.275 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:52.275 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:52.365 INFO:tasks.workunit.client.0.smithi142.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:44:52.373 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:44:52.373 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822668 1-' 2020-03-11T22:44:52.374 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:52.374 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:52.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=158913789964 2020-03-11T22:44:52.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-154618822668 1- 2-158913789964' 2020-03-11T22:44:52.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:52.504 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-154618822668 2020-03-11T22:44:52.504 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:52.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:52.505 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-154618822668 2020-03-11T22:44:52.505 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:52.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822668 2020-03-11T22:44:52.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 154618822668' 2020-03-11T22:44:52.506 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:52.506 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 154618822668 2020-03-11T22:44:52.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822669 -lt 154618822668 2020-03-11T22:44:52.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:52.873 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:44:52.873 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:52.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:52.874 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:44:52.874 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:52.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:44:52.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:44:52.875 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 2020-03-11T22:44:52.876 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:53.240 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 2020-03-11T22:44:53.240 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705679: unary operator expected 2020-03-11T22:44:53.240 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:53.241 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-158913789964 2020-03-11T22:44:53.241 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:53.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:53.242 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-158913789964 2020-03-11T22:44:53.242 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:53.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=158913789964 2020-03-11T22:44:53.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 158913789964' 2020-03-11T22:44:53.243 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 158913789964 2020-03-11T22:44:53.243 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 158913789964 -lt 158913789964 2020-03-11T22:44:53.590 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:53.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:54.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:44:54.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:54.042 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:54.042 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:54.042 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:54.042 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:54.043 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:54.043 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:54.380 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:54.441 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:54.441 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:54.441 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:54.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:54.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:44:54.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:44:54.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:44:54.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:54.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:44:54.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:54.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:54.929 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:54.929 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:54.960 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:54.960 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:54.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:54.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:55.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:55.264 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:55.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:55.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:55.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:55.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:55.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:55.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:55.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:55.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:55.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:55.710 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:55.710 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:56.153 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:56.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:56.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:56.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:56.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:56.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:56.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:56.355 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:56.355 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:56.355 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:56.356 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:56.356 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:56.356 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:56.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:56.688 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:56.688 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:56.688 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:57.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:57.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:57.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:57.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:57.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:57.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:57.127 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:57.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:57.128 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:57.568 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:57.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:57.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:57.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:57.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:57.969 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:57.970 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:57.971 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:57.971 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:57.971 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:57.971 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:57.971 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:57.971 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:58.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:58.304 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:58.305 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:58.305 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:58.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:44:58.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:58.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:58.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:58.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:58.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:58.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:58.748 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:58.748 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:59.185 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:59.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:59.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:59.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:59.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:59.986 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:59.987 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:59.988 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:59.988 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:59.988 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:59.989 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:59.989 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:59.989 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:00.328 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:00.329 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:00.329 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:00.329 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:00.779 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:45:00.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:00.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:00.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:00.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:00.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:00.781 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:00.781 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:00.781 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:01.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:01.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:01.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:45:01.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:01.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:45:02.821 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:02.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:02.822 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:02.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:02.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:02.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:02.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:02.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:03.167 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:03.167 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:03.167 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:03.168 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:03.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:45:03.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:03.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:03.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:03.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:03.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:03.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:03.611 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:03.611 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:04.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:04.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:04.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:45:04.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:04.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:45:07.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:07.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:07.250 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:07.251 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:07.251 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:07.251 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:07.251 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:07.252 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:07.598 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:07.599 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:07.599 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:07.599 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:08.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:45:08.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:08.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:08.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:08.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:08.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:08.042 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:08.042 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:08.042 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:08.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:08.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:08.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:45:08.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:08.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:45:14.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:14.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:14.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:14.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:14.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:14.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:14.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:14.889 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:15.232 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:15.233 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:15.233 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:15.233 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:15.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:45:15.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:15.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:15.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:15.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:15.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:15.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:15.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:15.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:16.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:16.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:16.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:45:16.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:16.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:45:23.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:23.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:23.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:23.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:23.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:23.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:23.418 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:23.418 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:23.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:23.761 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:23.762 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:23.762 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:24.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:45:24.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:24.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:24.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:24.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:24.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:24.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:24.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:24.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:24.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:24.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:24.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:45:24.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:45:25.080 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:45:25.080 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_fingerprint": "e56ab1ef-8a74-41c4-a177-661e9fcb9197", 2020-03-11T22:45:25.080 INFO:tasks.workunit.client.0.smithi142.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:45:25.081 INFO:tasks.workunit.client.0.smithi142.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:45:25.081 INFO:tasks.workunit.client.0.smithi142.stdout: "timestamp": "2020-03-11T22:45:25.076885+0000", 2020-03-11T22:45:25.081 INFO:tasks.workunit.client.0.smithi142.stdout: "tag": "", 2020-03-11T22:45:25.081 INFO:tasks.workunit.client.0.smithi142.stdout: "health": { 2020-03-11T22:45:25.081 INFO:tasks.workunit.client.0.smithi142.stdout: "status": "HEALTH_WARN", 2020-03-11T22:45:25.081 INFO:tasks.workunit.client.0.smithi142.stdout: "checks": { 2020-03-11T22:45:25.081 INFO:tasks.workunit.client.0.smithi142.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:45:25.082 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:25.082 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:45:25.082 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:45:25.082 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 17 2020-03-11T22:45:25.082 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.082 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [], 2020-03-11T22:45:25.082 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:45:25.083 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.083 INFO:tasks.workunit.client.0.smithi142.stdout: "OSD_DOWN": { 2020-03-11T22:45:25.083 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:25.083 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:45:25.083 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 osds down", 2020-03-11T22:45:25.083 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:45:25.083 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.084 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:45:25.084 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.084 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "osd.1 (root=default,host=smithi142) is down" 2020-03-11T22:45:25.084 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.084 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.084 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:45:25.084 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.085 INFO:tasks.workunit.client.0.smithi142.stdout: "PG_DEGRADED": { 2020-03-11T22:45:25.085 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:25.085 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:45:25.085 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:45:25.085 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 64 2020-03-11T22:45:25.086 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.086 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:45:25.086 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.086 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.0 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.086 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.086 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.087 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.087 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.087 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.087 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.2 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.087 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.087 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.087 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.3 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.088 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.088 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.088 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.4 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.088 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.088 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.088 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.5 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.088 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.089 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.089 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.6 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.089 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.089 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.089 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.7 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.089 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.089 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.090 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.8 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.090 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.090 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.090 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.9 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.090 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.090 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.a is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.b is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.c is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.091 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.092 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.092 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.d is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.092 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.092 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.092 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.e is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.092 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.092 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.093 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.f is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.093 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.093 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.093 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.10 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.093 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.093 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.093 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.11 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.094 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.094 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.094 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.12 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.094 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.094 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.094 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.13 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.094 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.095 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.095 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.14 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.095 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.095 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.095 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.15 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.095 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.095 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.096 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.16 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.096 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.096 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.096 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.17 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.096 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.096 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.096 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.18 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.097 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.097 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.097 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.19 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.097 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.097 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.097 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1a is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.097 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.098 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.098 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1b is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.098 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.098 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.098 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1c is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.098 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.099 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.099 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1d is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.099 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.099 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.099 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1e is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.099 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.099 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.100 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.1f is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:45:25.100 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.100 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.100 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:45:25.100 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.100 INFO:tasks.workunit.client.0.smithi142.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:45:25.100 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:45:25.101 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.102 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.102 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:45:25.102 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.102 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.102 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:45:25.102 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.102 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.103 INFO:tasks.workunit.client.0.smithi142.stdout: "mutes": [] 2020-03-11T22:45:25.103 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.103 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_first_committed": 1, 2020-03-11T22:45:25.103 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_last_committed": 1, 2020-03-11T22:45:25.103 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap": { 2020-03-11T22:45:25.103 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:45:25.103 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:45:25.104 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:43:11.901399Z", 2020-03-11T22:45:25.104 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:43:11.901399Z", 2020-03-11T22:45:25.104 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release": 15, 2020-03-11T22:45:25.104 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:45:25.104 INFO:tasks.workunit.client.0.smithi142.stdout: "features": { 2020-03-11T22:45:25.104 INFO:tasks.workunit.client.0.smithi142.stdout: "persistent": [ 2020-03-11T22:45:25.104 INFO:tasks.workunit.client.0.smithi142.stdout: "kraken", 2020-03-11T22:45:25.105 INFO:tasks.workunit.client.0.smithi142.stdout: "luminous", 2020-03-11T22:45:25.105 INFO:tasks.workunit.client.0.smithi142.stdout: "mimic", 2020-03-11T22:45:25.105 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap-prune", 2020-03-11T22:45:25.105 INFO:tasks.workunit.client.0.smithi142.stdout: "nautilus", 2020-03-11T22:45:25.105 INFO:tasks.workunit.client.0.smithi142.stdout: "octopus" 2020-03-11T22:45:25.105 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.105 INFO:tasks.workunit.client.0.smithi142.stdout: "optional": [] 2020-03-11T22:45:25.106 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.106 INFO:tasks.workunit.client.0.smithi142.stdout: "mons": [ 2020-03-11T22:45:25.106 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.106 INFO:tasks.workunit.client.0.smithi142.stdout: "rank": 0, 2020-03-11T22:45:25.106 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "a", 2020-03-11T22:45:25.106 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:45:25.106 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.107 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.107 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.107 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:45:25.107 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 0 2020-03-11T22:45:25.107 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.107 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.107 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.108 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:45:25.108 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:45:25.108 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 0, 2020-03-11T22:45:25.108 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 0 2020-03-11T22:45:25.108 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.108 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.108 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.109 INFO:tasks.workunit.client.0.smithi142.stdout: "quorum": [ 2020-03-11T22:45:25.109 INFO:tasks.workunit.client.0.smithi142.stdout: 0 2020-03-11T22:45:25.109 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.109 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap": { 2020-03-11T22:45:25.109 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 43, 2020-03-11T22:45:25.109 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:45:25.109 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:43:12.586713+0000", 2020-03-11T22:45:25.110 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:44:18.977612+0000", 2020-03-11T22:45:25.110 INFO:tasks.workunit.client.0.smithi142.stdout: "last_up_change": "2020-03-11T22:44:14.647540+0000", 2020-03-11T22:45:25.110 INFO:tasks.workunit.client.0.smithi142.stdout: "last_in_change": "2020-03-11T22:44:16.361638+0000", 2020-03-11T22:45:25.110 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:45:25.110 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_num": 5800384, 2020-03-11T22:45:25.110 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_set": [ 2020-03-11T22:45:25.110 INFO:tasks.workunit.client.0.smithi142.stdout: "nodown", 2020-03-11T22:45:25.111 INFO:tasks.workunit.client.0.smithi142.stdout: "noin", 2020-03-11T22:45:25.111 INFO:tasks.workunit.client.0.smithi142.stdout: "noout", 2020-03-11T22:45:25.111 INFO:tasks.workunit.client.0.smithi142.stdout: "pglog_hardlimit", 2020-03-11T22:45:25.111 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snapdirs", 2020-03-11T22:45:25.111 INFO:tasks.workunit.client.0.smithi142.stdout: "recovery_deletes", 2020-03-11T22:45:25.111 INFO:tasks.workunit.client.0.smithi142.stdout: "sortbitwise" 2020-03-11T22:45:25.111 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.112 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_version": 7, 2020-03-11T22:45:25.112 INFO:tasks.workunit.client.0.smithi142.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:45:25.112 INFO:tasks.workunit.client.0.smithi142.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:45:25.112 INFO:tasks.workunit.client.0.smithi142.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:45:25.112 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_snapshot": "", 2020-03-11T22:45:25.112 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_max": 1, 2020-03-11T22:45:25.112 INFO:tasks.workunit.client.0.smithi142.stdout: "max_osd": 3, 2020-03-11T22:45:25.113 INFO:tasks.workunit.client.0.smithi142.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:45:25.113 INFO:tasks.workunit.client.0.smithi142.stdout: "min_compat_client": "jewel", 2020-03-11T22:45:25.113 INFO:tasks.workunit.client.0.smithi142.stdout: "require_osd_release": "octopus", 2020-03-11T22:45:25.113 INFO:tasks.workunit.client.0.smithi142.stdout: "pools": [ 2020-03-11T22:45:25.113 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.113 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:45:25.113 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_name": "test", 2020-03-11T22:45:25.114 INFO:tasks.workunit.client.0.smithi142.stdout: "create_time": "2020-03-11T22:43:33.821631+0000", 2020-03-11T22:45:25.114 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": 1, 2020-03-11T22:45:25.114 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_names": "hashpspool", 2020-03-11T22:45:25.114 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:45:25.114 INFO:tasks.workunit.client.0.smithi142.stdout: "size": 3, 2020-03-11T22:45:25.114 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 2, 2020-03-11T22:45:25.114 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_rule": 0, 2020-03-11T22:45:25.115 INFO:tasks.workunit.client.0.smithi142.stdout: "object_hash": 2, 2020-03-11T22:45:25.115 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:45:25.115 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num": 32, 2020-03-11T22:45:25.115 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num": 1, 2020-03-11T22:45:25.115 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num_target": 32, 2020-03-11T22:45:25.115 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_target": 32, 2020-03-11T22:45:25.115 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_pending": 32, 2020-03-11T22:45:25.116 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pg_merge_meta": { 2020-03-11T22:45:25.116 INFO:tasks.workunit.client.0.smithi142.stdout: "source_pgid": "0.0", 2020-03-11T22:45:25.116 INFO:tasks.workunit.client.0.smithi142.stdout: "ready_epoch": 0, 2020-03-11T22:45:25.116 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_started": 0, 2020-03-11T22:45:25.116 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_clean": 0, 2020-03-11T22:45:25.116 INFO:tasks.workunit.client.0.smithi142.stdout: "source_version": "0'0", 2020-03-11T22:45:25.116 INFO:tasks.workunit.client.0.smithi142.stdout: "target_version": "0'0" 2020-03-11T22:45:25.117 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.117 INFO:tasks.workunit.client.0.smithi142.stdout: "last_change": "42", 2020-03-11T22:45:25.117 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend": "0", 2020-03-11T22:45:25.117 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:45:25.117 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_preluminous": "42", 2020-03-11T22:45:25.117 INFO:tasks.workunit.client.0.smithi142.stdout: "auid": 0, 2020-03-11T22:45:25.117 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:45:25.118 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_seq": 0, 2020-03-11T22:45:25.118 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_epoch": 0, 2020-03-11T22:45:25.118 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_snaps": [], 2020-03-11T22:45:25.118 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps": "[]", 2020-03-11T22:45:25.118 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_bytes": 0, 2020-03-11T22:45:25.118 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_objects": 0, 2020-03-11T22:45:25.118 INFO:tasks.workunit.client.0.smithi142.stdout: "tiers": [], 2020-03-11T22:45:25.119 INFO:tasks.workunit.client.0.smithi142.stdout: "tier_of": -1, 2020-03-11T22:45:25.119 INFO:tasks.workunit.client.0.smithi142.stdout: "read_tier": -1, 2020-03-11T22:45:25.119 INFO:tasks.workunit.client.0.smithi142.stdout: "write_tier": -1, 2020-03-11T22:45:25.119 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_mode": "none", 2020-03-11T22:45:25.119 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_bytes": 0, 2020-03-11T22:45:25.119 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_objects": 0, 2020-03-11T22:45:25.119 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:45:25.120 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:45:25.120 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:45:25.120 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_flush_age": 0, 2020-03-11T22:45:25.120 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_evict_age": 0, 2020-03-11T22:45:25.120 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profile": "", 2020-03-11T22:45:25.121 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_params": { 2020-03-11T22:45:25.121 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "none" 2020-03-11T22:45:25.121 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.121 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_period": 0, 2020-03-11T22:45:25.121 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_count": 0, 2020-03-11T22:45:25.121 INFO:tasks.workunit.client.0.smithi142.stdout: "use_gmt_hitset": true, 2020-03-11T22:45:25.121 INFO:tasks.workunit.client.0.smithi142.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:45:25.122 INFO:tasks.workunit.client.0.smithi142.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:45:25.122 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:45:25.122 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:45:25.122 INFO:tasks.workunit.client.0.smithi142.stdout: "grade_table": [], 2020-03-11T22:45:25.122 INFO:tasks.workunit.client.0.smithi142.stdout: "stripe_width": 0, 2020-03-11T22:45:25.122 INFO:tasks.workunit.client.0.smithi142.stdout: "expected_num_objects": 0, 2020-03-11T22:45:25.122 INFO:tasks.workunit.client.0.smithi142.stdout: "fast_read": false, 2020-03-11T22:45:25.123 INFO:tasks.workunit.client.0.smithi142.stdout: "options": {}, 2020-03-11T22:45:25.123 INFO:tasks.workunit.client.0.smithi142.stdout: "application_metadata": {} 2020-03-11T22:45:25.123 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.123 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.123 INFO:tasks.workunit.client.0.smithi142.stdout: "osds": [ 2020-03-11T22:45:25.123 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.123 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:45:25.124 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "65f689f5-c5c4-499f-8f76-b5d7d974b0d8", 2020-03-11T22:45:25.124 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:45:25.124 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:45:25.124 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:45:25.124 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:45:25.124 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 5, 2020-03-11T22:45:25.124 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 30, 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 36, 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 42, 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 31, 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.125 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.126 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:45:25.126 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.126 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.126 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.126 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.126 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:45:25.126 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.127 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.127 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.127 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.127 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:45:25.127 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.127 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.127 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.128 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:45:25.128 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.128 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.128 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.128 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.128 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:45:25.128 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.129 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:45:25.130 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.130 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.130 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.130 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.130 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:45:25.130 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.130 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.131 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.131 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.131 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:25.131 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.131 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.131 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.131 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:45:25.132 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.132 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.132 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.132 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.132 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:45:25.132 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39340 2020-03-11T22:45:25.132 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.133 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.133 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.133 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6803/39340", 2020-03-11T22:45:25.133 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6805/39340", 2020-03-11T22:45:25.133 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39340", 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39340", 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.134 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:45:25.135 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "e79d27b4-6fc2-43ed-90f7-84fe096e508b", 2020-03-11T22:45:25.135 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 0, 2020-03-11T22:45:25.135 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:45:25.135 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:45:25.135 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:45:25.135 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 0, 2020-03-11T22:45:25.135 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 0, 2020-03-11T22:45:25.136 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 9, 2020-03-11T22:45:25.136 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 22, 2020-03-11T22:45:25.136 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 32, 2020-03-11T22:45:25.136 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:45:25.136 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:45:25.136 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.136 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.137 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.137 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:25.137 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.137 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.137 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.137 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.137 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:25.138 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.138 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.138 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.138 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.138 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:45:25.138 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.138 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:25.139 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.140 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.140 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.140 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.140 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:25.140 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.140 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.140 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.141 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:25.141 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.141 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.141 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.141 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.141 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:25.141 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.142 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.142 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.142 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.142 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:25.142 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.142 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.142 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.143 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:25.143 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.143 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.143 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.143 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.143 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:25.143 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 34873 2020-03-11T22:45:25.144 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.144 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.144 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.144 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/34873", 2020-03-11T22:45:25.144 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/34873", 2020-03-11T22:45:25.144 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34873", 2020-03-11T22:45:25.144 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34873", 2020-03-11T22:45:25.145 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:45:25.145 INFO:tasks.workunit.client.0.smithi142.stdout: "exists" 2020-03-11T22:45:25.145 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.145 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.145 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.145 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:45:25.145 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "19594912-e0ba-4ab5-9da4-a2f9818068ee", 2020-03-11T22:45:25.146 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:45:25.146 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:45:25.146 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:45:25.146 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:45:25.146 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 14, 2020-03-11T22:45:25.146 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 30, 2020-03-11T22:45:25.146 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 37, 2020-03-11T22:45:25.147 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 0, 2020-03-11T22:45:25.147 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 33, 2020-03-11T22:45:25.147 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:45:25.147 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:45:25.147 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.147 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.147 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.148 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:25.148 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.148 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.148 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.148 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.148 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:25.148 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.149 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.149 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.149 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.149 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:45:25.149 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.149 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.149 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.150 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:25.150 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.150 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.150 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.150 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.150 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:25.150 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.151 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:25.152 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.152 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.152 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.152 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.152 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:25.152 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.152 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.153 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.153 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.153 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:25.153 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:45:25.153 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.153 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:45:25.154 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:25.154 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.154 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.154 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.154 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:45:25.154 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:25.154 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 39534 2020-03-11T22:45:25.155 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.155 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.155 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.155 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/39534", 2020-03-11T22:45:25.155 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/39534", 2020-03-11T22:45:25.155 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39534", 2020-03-11T22:45:25.155 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39534", 2020-03-11T22:45:25.156 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:45:25.156 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:45:25.156 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:45:25.156 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.156 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.156 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.156 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_xinfo": [ 2020-03-11T22:45:25.157 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.157 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:45:25.157 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:44:08.519711+0000", 2020-03-11T22:45:25.157 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:45:25.157 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:45:25.157 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:45:25.157 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:45:25.158 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:19.534845+0000", 2020-03-11T22:45:25.158 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:45:25.158 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.158 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.158 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:45:25.158 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:44:09.043221+0000", 2020-03-11T22:45:25.158 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:45:25.159 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:45:25.159 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:45:25.159 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:45:25.159 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:25.625940+0000", 2020-03-11T22:45:25.159 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:45:25.159 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.159 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.160 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:45:25.160 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:44:09.557846+0000", 2020-03-11T22:45:25.160 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:45:25.160 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:45:25.160 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:45:25.160 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:45:25.160 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:31.707590+0000", 2020-03-11T22:45:25.161 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:45:25.161 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.161 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.161 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap": [], 2020-03-11T22:45:25.161 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap_items": [], 2020-03-11T22:45:25.161 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_temp": [], 2020-03-11T22:45:25.161 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_temp": [], 2020-03-11T22:45:25.162 INFO:tasks.workunit.client.0.smithi142.stdout: "blacklist": {}, 2020-03-11T22:45:25.162 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profiles": { 2020-03-11T22:45:25.162 INFO:tasks.workunit.client.0.smithi142.stdout: "default": { 2020-03-11T22:45:25.162 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:45:25.162 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "2", 2020-03-11T22:45:25.162 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:45:25.162 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van" 2020-03-11T22:45:25.163 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.163 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.163 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps_queue": [], 2020-03-11T22:45:25.163 INFO:tasks.workunit.client.0.smithi142.stdout: "new_removed_snaps": [], 2020-03-11T22:45:25.163 INFO:tasks.workunit.client.0.smithi142.stdout: "new_purged_snaps": [], 2020-03-11T22:45:25.163 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_node_flags": {}, 2020-03-11T22:45:25.163 INFO:tasks.workunit.client.0.smithi142.stdout: "device_class_flags": {} 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_metadata": [ 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6804/39340,v1:127.0.0.1:6805/39340]", 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:45:25.164 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:45:25.165 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:25.165 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:25.165 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:25.165 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:25.165 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:25.165 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:25.165 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:25.166 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:45:25.166 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:25.166 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:25.166 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:25.166 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:25.166 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:45:25.166 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:25.167 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:25.167 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:25.167 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:45:25.167 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:45:25.167 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:25.167 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:45:25.168 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:45:25.168 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:25.168 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:45:25.168 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6802/39340,v1:127.0.0.1:6803/39340]", 2020-03-11T22:45:25.168 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:45:25.168 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39340,v1:127.0.0.1:6809/39340]", 2020-03-11T22:45:25.168 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39340,v1:127.0.0.1:6807/39340]", 2020-03-11T22:45:25.169 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:45:25.169 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:45:25.169 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:25.169 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:25.169 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:25.169 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:45:25.169 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:25.170 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:25.170 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:45:25.170 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:45:25.170 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:25.170 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:45:25.170 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.170 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.171 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:45:25.171 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:45:25.171 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/34873,v1:127.0.0.1:6813/34873]", 2020-03-11T22:45:25.171 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:45:25.171 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:45:25.171 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:25.171 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:25.172 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:25.172 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:25.172 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:25.172 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:25.172 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:25.172 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:45:25.173 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:25.173 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:25.173 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:25.173 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:25.173 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:45:25.173 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:25.173 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:25.174 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:25.174 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:45:25.174 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:45:25.174 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:25.174 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:45:25.174 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:45:25.174 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:25.175 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:45:25.175 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/34873,v1:127.0.0.1:6811/34873]", 2020-03-11T22:45:25.175 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:45:25.175 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34873,v1:127.0.0.1:6817/34873]", 2020-03-11T22:45:25.175 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34873,v1:127.0.0.1:6815/34873]", 2020-03-11T22:45:25.175 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:45:25.175 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:45:25.176 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:25.176 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:25.176 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:25.176 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:45:25.176 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:25.176 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:25.176 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:45:25.177 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/39534,v1:127.0.0.1:6813/39534]", 2020-03-11T22:45:25.178 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:45:25.178 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:45:25.178 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:25.178 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:25.178 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:25.178 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:25.178 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:25.179 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:25.179 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:25.179 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:45:25.179 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:25.179 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:25.179 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:25.179 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:25.180 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:45:25.180 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:25.180 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:25.180 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:25.180 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:45:25.180 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:45:25.180 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:25.181 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:45:25.181 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:45:25.181 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:25.181 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:45:25.181 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/39534,v1:127.0.0.1:6811/39534]", 2020-03-11T22:45:25.181 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:45:25.181 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39534,v1:127.0.0.1:6817/39534]", 2020-03-11T22:45:25.182 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39534,v1:127.0.0.1:6815/39534]", 2020-03-11T22:45:25.182 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:45:25.182 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:45:25.182 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:25.182 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:25.182 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:25.182 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:45:25.183 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:25.183 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:25.183 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:45:25.183 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:45:25.183 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:25.183 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:45:25.183 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.184 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.184 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_first_committed": 1, 2020-03-11T22:45:25.184 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_last_committed": 43, 2020-03-11T22:45:25.184 INFO:tasks.workunit.client.0.smithi142.stdout: "crushmap": { 2020-03-11T22:45:25.184 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": [ 2020-03-11T22:45:25.184 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.184 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:45:25.185 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.0", 2020-03-11T22:45:25.185 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:45:25.185 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.185 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.185 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:45:25.185 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.1", 2020-03-11T22:45:25.186 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:45:25.186 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.186 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.186 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:45:25.186 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.2", 2020-03-11T22:45:25.186 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:45:25.186 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.187 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.187 INFO:tasks.workunit.client.0.smithi142.stdout: "types": [ 2020-03-11T22:45:25.187 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.187 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 0, 2020-03-11T22:45:25.187 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd" 2020-03-11T22:45:25.187 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.188 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.188 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:45:25.188 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "host" 2020-03-11T22:45:25.188 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.188 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.188 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 2, 2020-03-11T22:45:25.188 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "chassis" 2020-03-11T22:45:25.189 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.189 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.189 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 3, 2020-03-11T22:45:25.189 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "rack" 2020-03-11T22:45:25.189 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.189 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.189 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 4, 2020-03-11T22:45:25.190 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "row" 2020-03-11T22:45:25.190 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.190 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.190 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 5, 2020-03-11T22:45:25.190 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pdu" 2020-03-11T22:45:25.190 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.190 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 6, 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pod" 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 7, 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "room" 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.191 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.192 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 8, 2020-03-11T22:45:25.192 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "datacenter" 2020-03-11T22:45:25.192 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.192 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.192 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 9, 2020-03-11T22:45:25.192 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "zone" 2020-03-11T22:45:25.192 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.193 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.193 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 10, 2020-03-11T22:45:25.193 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "region" 2020-03-11T22:45:25.193 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.193 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.193 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:45:25.193 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "root" 2020-03-11T22:45:25.194 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.194 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.194 INFO:tasks.workunit.client.0.smithi142.stdout: "buckets": [ 2020-03-11T22:45:25.194 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.194 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -1, 2020-03-11T22:45:25.194 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default", 2020-03-11T22:45:25.194 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:45:25.195 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:45:25.196 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:45:25.196 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.196 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.196 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.196 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.196 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -2, 2020-03-11T22:45:25.196 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default~hdd", 2020-03-11T22:45:25.197 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:45:25.197 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:45:25.197 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:45:25.197 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:45:25.197 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:45:25.197 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:45:25.197 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.198 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:45:25.198 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:45:25.198 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:45:25.198 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.198 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.198 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.198 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.199 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:45:25.199 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142", 2020-03-11T22:45:25.199 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:45:25.199 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:45:25.199 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:45:25.199 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.200 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.201 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:45:25.201 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:45:25.201 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:45:25.201 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.201 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.201 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:45:25.201 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:45:25.202 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:45:25.202 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.202 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.202 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.202 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.202 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:45:25.202 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142~hdd", 2020-03-11T22:45:25.203 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:45:25.203 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:45:25.203 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:45:25.203 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:45:25.203 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:45:25.203 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:45:25.203 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.204 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:45:25.204 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:45:25.204 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:45:25.204 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.204 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.204 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:45:25.204 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:45:25.205 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:45:25.205 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.205 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.205 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:45:25.205 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:45:25.205 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:45:25.205 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.206 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.206 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.206 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.206 INFO:tasks.workunit.client.0.smithi142.stdout: "rules": [ 2020-03-11T22:45:25.206 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.206 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 0, 2020-03-11T22:45:25.206 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "replicated_rule", 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 0, 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 1, 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 10, 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:45:25.207 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:45:25.208 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:45:25.208 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.208 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.208 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_firstn", 2020-03-11T22:45:25.208 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:45:25.208 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:45:25.208 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.209 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.209 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:45:25.209 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.209 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:45:25.209 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.209 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.209 INFO:tasks.workunit.client.0.smithi142.stdout: "tunables": { 2020-03-11T22:45:25.210 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_tries": 0, 2020-03-11T22:45:25.210 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:45:25.210 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_total_tries": 50, 2020-03-11T22:45:25.210 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:45:25.210 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:45:25.210 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_stable": 1, 2020-03-11T22:45:25.210 INFO:tasks.workunit.client.0.smithi142.stdout: "straw_calc_version": 1, 2020-03-11T22:45:25.211 INFO:tasks.workunit.client.0.smithi142.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:45:25.211 INFO:tasks.workunit.client.0.smithi142.stdout: "profile": "jewel", 2020-03-11T22:45:25.211 INFO:tasks.workunit.client.0.smithi142.stdout: "optimal_tunables": 1, 2020-03-11T22:45:25.211 INFO:tasks.workunit.client.0.smithi142.stdout: "legacy_tunables": 0, 2020-03-11T22:45:25.211 INFO:tasks.workunit.client.0.smithi142.stdout: "minimum_required_version": "jewel", 2020-03-11T22:45:25.211 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables": 1, 2020-03-11T22:45:25.211 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables2": 1, 2020-03-11T22:45:25.212 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v2_rules": 0, 2020-03-11T22:45:25.212 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables3": 1, 2020-03-11T22:45:25.212 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v3_rules": 0, 2020-03-11T22:45:25.212 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v4_buckets": 1, 2020-03-11T22:45:25.212 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables5": 1, 2020-03-11T22:45:25.212 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v5_rules": 0 2020-03-11T22:45:25.212 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.213 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_args": {} 2020-03-11T22:45:25.213 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.213 INFO:tasks.workunit.client.0.smithi142.stdout: "fsmap": { 2020-03-11T22:45:25.213 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:45:25.213 INFO:tasks.workunit.client.0.smithi142.stdout: "default_fscid": -1, 2020-03-11T22:45:25.213 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": { 2020-03-11T22:45:25.213 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": {}, 2020-03-11T22:45:25.214 INFO:tasks.workunit.client.0.smithi142.stdout: "ro_compat": {}, 2020-03-11T22:45:25.214 INFO:tasks.workunit.client.0.smithi142.stdout: "incompat": { 2020-03-11T22:45:25.214 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_1": "base v0.20", 2020-03-11T22:45:25.214 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:45:25.214 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:45:25.214 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:45:25.214 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:45:25.215 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:45:25.215 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_8": "no anchor table", 2020-03-11T22:45:25.215 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_9": "file layout v2", 2020-03-11T22:45:25.215 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:45:25.215 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.215 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.215 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_flags": { 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: "enable_multiple": false, 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: "ever_enabled_multiple": false 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: "standbys": [], 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: "filesystems": [] 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:45:25.216 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:45:25.217 INFO:tasks.workunit.client.0.smithi142.stdout: "auth": { 2020-03-11T22:45:25.217 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:45:25.217 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 5, 2020-03-11T22:45:25.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_secrets": 3 2020-03-11T22:45:25.217 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 32, 2020-03-11T22:45:25.217 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_active": 32, 2020-03-11T22:45:25.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_unknown": 0, 2020-03-11T22:45:25.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osd": 3, 2020-03-11T22:45:25.218 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_sum": { 2020-03-11T22:45:25.218 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:45:25.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 47947, 2020-03-11T22:45:25.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:45:25.218 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:45:25.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:45:25.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:25.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:45:25.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:45:25.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:25.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:25.219 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:45:25.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:45:25.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:45:25.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:45:25.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:45:25.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:45:25.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:25.220 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:25.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:25.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:25.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:25.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:25.221 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:45:25.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:25.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:25.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:45:25.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:45:25.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:45:25.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:45:25.222 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:45:25.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:25.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:25.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:25.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:25.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:25.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:25.223 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:25.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:25.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:25.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:45:25.224 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:45:25.224 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.224 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:45:25.224 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:45:25.225 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:45:25.225 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:45:25.225 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 0, 2020-03-11T22:45:25.225 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 0, 2020-03-11T22:45:25.225 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:45:25.225 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:25.225 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:45:25.226 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:45:25.226 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:45:25.226 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.226 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 513, 2020-03-11T22:45:25.226 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 513, 2020-03-11T22:45:25.226 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 64, 2020-03-11T22:45:25.226 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 64, 2020-03-11T22:45:25.227 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 0 2020-03-11T22:45:25.227 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.227 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum": { 2020-03-11T22:45:25.227 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:45:25.227 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:45:25.227 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 65, 2020-03-11T22:45:25.227 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3180288, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 34240, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 6, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145721, 2020-03-11T22:45:25.228 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311392512, 2020-03-11T22:45:25.229 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:45:25.229 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:45:25.229 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318865932288, 2020-03-11T22:45:25.229 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:25.229 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 35061760, 2020-03-11T22:45:25.229 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 433460, 2020-03-11T22:45:25.229 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:45:25.230 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:25.230 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:45:25.230 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 6916, 2020-03-11T22:45:25.230 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221218556 2020-03-11T22:45:25.230 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.230 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:45:25.230 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:45:25.231 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:25.232 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:25.232 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:25.232 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:45:25.232 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.232 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:45:25.232 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:45:25.232 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.233 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum_by_class": { 2020-03-11T22:45:25.233 INFO:tasks.workunit.client.0.smithi142.stdout: "hdd": { 2020-03-11T22:45:25.233 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:45:25.233 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:45:25.233 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 65, 2020-03-11T22:45:25.233 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:45:25.233 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:25.234 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:25.234 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:45:25.234 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3180288, 2020-03-11T22:45:25.234 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 34240, 2020-03-11T22:45:25.234 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 6, 2020-03-11T22:45:25.234 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145721, 2020-03-11T22:45:25.235 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311392512, 2020-03-11T22:45:25.235 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:45:25.235 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:45:25.235 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318865932288, 2020-03-11T22:45:25.235 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:25.235 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 35061760, 2020-03-11T22:45:25.235 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 433460, 2020-03-11T22:45:25.236 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:45:25.236 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:25.236 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:45:25.236 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 6916, 2020-03-11T22:45:25.236 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221218556 2020-03-11T22:45:25.236 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.236 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.237 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:45:25.238 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:25.238 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:25.238 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:25.238 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:45:25.238 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.238 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:45:25.238 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:45:25.239 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.239 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.239 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_stats": [ 2020-03-11T22:45:25.239 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.239 INFO:tasks.workunit.client.0.smithi142.stdout: "poolid": 1, 2020-03-11T22:45:25.239 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 32, 2020-03-11T22:45:25.239 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:45:25.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 47947, 2020-03-11T22:45:25.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 100, 2020-03-11T22:45:25.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 0, 2020-03-11T22:45:25.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 300, 2020-03-11T22:45:25.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:25.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:45:25.240 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 100, 2020-03-11T22:45:25.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:25.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:25.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 100, 2020-03-11T22:45:25.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:45:25.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:45:25.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:45:25.241 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:25.242 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:45:25.243 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:25.243 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:25.243 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:45:25.243 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:45:25.243 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:45:25.243 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:45:25.243 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:45:25.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:25.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:25.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:25.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:25.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:25.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:25.244 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 19529728, 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 142522, 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 513, 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 513, 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 64, 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 64, 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 3 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_stats": [ 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 154618822675 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.249 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.249 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:45:25.249 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 38654705679 2020-03-11T22:45:25.249 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.249 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.249 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:45:25.249 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 158913789971 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_state": [ 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: "state": "active+undersized+degraded", 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 32 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.250 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.251 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_osd": [ 2020-03-11T22:45:25.251 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.251 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:45:25.251 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:45:25.251 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 32, 2020-03-11T22:45:25.251 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:25.251 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.252 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.252 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:45:25.252 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:45:25.252 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 0, 2020-03-11T22:45:25.252 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:25.252 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.252 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.253 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:45:25.253 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 32, 2020-03-11T22:45:25.253 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 32, 2020-03-11T22:45:25.253 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:25.253 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.253 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.253 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": [ 2020-03-11T22:45:25.254 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:45:25.254 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:45:25.254 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": {} 2020-03-11T22:45:25.254 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.254 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:45:25.254 INFO:tasks.workunit.client.0.smithi142.stdout: "servicemap": { 2020-03-11T22:45:25.254 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:45:25.255 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:43:12.587022+0000", 2020-03-11T22:45:25.255 INFO:tasks.workunit.client.0.smithi142.stdout: "services": {} 2020-03-11T22:45:25.255 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:45:25.255 INFO:tasks.workunit.client.0.smithi142.stdout: "paxos": { 2020-03-11T22:45:25.255 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:45:25.255 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 391, 2020-03-11T22:45:25.255 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pn": 0, 2020-03-11T22:45:25.256 INFO:tasks.workunit.client.0.smithi142.stdout: "accepted_pn": 0 2020-03-11T22:45:25.256 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:45:25.256 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:45:25.257 INFO:tasks.workunit.client.0.smithi142.stderr:report 983164979 2020-03-11T22:45:25.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:45:25.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:45:25.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:25.258 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery 2020-03-11T22:45:25.410 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:25.411 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282996+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.411 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281033+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.411 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281891+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.411 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282998+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.411 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281529+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.411 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279177+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.412 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279169+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.412 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281526+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.412 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282095+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.412 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281173+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.412 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280241+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.412 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279827+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.413 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281136+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.413 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281718+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.413 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280838+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.413 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280668+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.413 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280205+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.413 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280456+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.413 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279651+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.414 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280405+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.414 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:32.641933+0000 30'100 43:134 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.414 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281745+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.414 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282114+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.414 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279429+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.414 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279199+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.414 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279163+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.415 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281429+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.415 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279830+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.415 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280031+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.415 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280292+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.415 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:44:19.279863+0000 43'101 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.415 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280584+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.416 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:45:25.416 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:25.416 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:45:25.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:45:25.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:45:25.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:45:25.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:45:25.422 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:25.422 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:25.422 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:45:25.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:45:25.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:45:25.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:45:25.513 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:45:25.520 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:45:25.521 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:45:25.521 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:45:25.521 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:45:25.522 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:25.522 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:25.522 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:45:25.522 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:45:25.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:45:25.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:45:25.613 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:45:25.619 INFO:tasks.workunit.client.0.smithi142.stdout:revive divergent 1 2020-03-11T22:45:25.619 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:45:25.620 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:25.940 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:25.940 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282996+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.941 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281033+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.941 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281891+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.941 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282998+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.941 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281529+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.942 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279177+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.942 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279169+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.942 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281526+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.942 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282095+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.943 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281173+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.943 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280241+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.943 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279827+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.943 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281136+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.944 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281718+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.944 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280838+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.944 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280668+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.944 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280205+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.944 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280456+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.945 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279651+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.945 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280405+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.945 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:32.641933+0000 30'100 43:134 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.945 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281745+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.946 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282114+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.946 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279429+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.946 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279199+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.946 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279163+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.947 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281429+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.947 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279830+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.947 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280031+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.947 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280292+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.948 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:44:19.279863+0000 43'101 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.948 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280584+0000 30'100 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:25.948 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:45:25.948 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:25.949 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:45:25.950 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:45:26.551 INFO:tasks.workunit.client.0.smithi142.stderr:noup is set 2020-03-11T22:45:26.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:45:26.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:26.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:26.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:45:26.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:26.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:45:26.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:26.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:26.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:26.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:26.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:45:26.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:45:26.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:26.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:26.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:26.565 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:26.565 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:26.565 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:26.565 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:26.565 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:26.565 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:45:26.566 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:45:26.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:45:26.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:26.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:26.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:26.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:26.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:26.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:26.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:26.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:26.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:45:26.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:45:26.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:26.567 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:45:26.568 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:45:26.568 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:45:26.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:26.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:26.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:26.589 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:26.586+0000 7f092d08cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:26.590 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:26.588+0000 7f092d08cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:26.592 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:26.589+0000 7f092d08cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:26.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:45:26.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:45:27.107 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:27.104+0000 7f092d08cec0 -1 Falling back to public interface 2020-03-11T22:45:27.924 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:27.921+0000 7f092d08cec0 -1 osd.1 30 log_to_monitors {default=true} 2020-03-11T22:45:31.935 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery divergent 2020-03-11T22:45:31.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:45:31.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:32.267 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:32.267 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282996+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.268 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281033+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.268 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281891+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.268 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282998+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.268 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281529+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.268 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279177+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.268 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279169+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.269 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281526+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.269 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282095+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.269 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281173+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.269 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280241+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.269 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279827+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.269 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281136+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.270 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281718+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.270 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280838+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.270 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280668+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.270 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280205+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.270 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280456+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.270 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279651+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.270 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280405+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.271 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:32.641933+0000 30'100 43:135 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.271 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281745+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.271 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282114+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.271 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279429+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.271 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279199+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.271 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279163+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.272 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281429+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.272 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279830+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.272 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280031+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.272 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280292+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.272 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:44:19.279863+0000 43'101 43:134 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.272 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280584+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:32.272 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:45:32.273 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:32.273 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:45:32.277 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:45:32.278 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:45:32.278 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:45:32.278 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:32.278 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:32.278 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:45:32.278 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:45:32.279 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:45:32.279 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:45:32.369 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:45:32.376 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:45:33.236 INFO:tasks.workunit.client.0.smithi142.stderr:noup is unset 2020-03-11T22:45:33.245 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:45:33.245 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:33.245 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:33.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:33.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:33.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:33.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:33.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:33.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:33.247 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:45:33.607 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 36 up_thru 42 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/39340,v1:127.0.0.1:6803/39340] [v2:127.0.0.1:6804/39340,v1:127.0.0.1:6805/39340] exists,up 65f689f5-c5c4-499f-8f76-b5d7d974b0d8 2020-03-11T22:45:33.607 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:45:33.607 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:33.607 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:33.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:33.609 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:33.609 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:33.609 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:33.947 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [9,30) [v2:127.0.0.1:6818/42063,v1:127.0.0.1:6819/42063] [v2:127.0.0.1:6820/42063,v1:127.0.0.1:6821/42063] exists,up e79d27b4-6fc2-43ed-90f7-84fe096e508b 2020-03-11T22:45:33.947 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:45:33.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:33.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:33.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:33.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:45:33.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:33.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:33.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:33.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:33.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:33.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:33.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:33.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:34.287 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 33 last_clean_interval [14,30) [v2:127.0.0.1:6810/39534,v1:127.0.0.1:6811/39534] [v2:127.0.0.1:6812/39534,v1:127.0.0.1:6813/39534] exists,up 19594912-e0ba-4ab5-9da4-a2f9818068ee 2020-03-11T22:45:34.287 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:34.287 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:34.287 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:34.287 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:34.615 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:34.615 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282996+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.615 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281033+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.615 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281891+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.615 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282998+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.616 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281529+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.616 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279177+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.616 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279169+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.616 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281526+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.616 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282095+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.616 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281173+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.617 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280241+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.617 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279827+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.617 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281136+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.617 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281718+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.617 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280838+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.617 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280668+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.617 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280205+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.618 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280456+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.618 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279651+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.618 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280405+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.618 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:32.641933+0000 30'100 43:135 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.618 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281745+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.618 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282114+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.618 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279429+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.619 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279199+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.619 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279163+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.619 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281429+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.619 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279830+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.619 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280031+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.619 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280292+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.620 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:44:19.279863+0000 43'101 43:134 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.620 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280584+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.620 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:45:34.620 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:34.621 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:45:34.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:45:34.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:34.625 INFO:tasks.workunit.client.0.smithi142.stdout:wait for peering 2020-03-11T22:45:34.954 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:34.954 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282996+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.955 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281033+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.955 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281891+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.955 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282998+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.955 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281529+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.955 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279177+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.955 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279169+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.955 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281526+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.956 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282095+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.956 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281173+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.956 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280241+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.956 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279827+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.956 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281136+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.956 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281718+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.957 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280838+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.957 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280668+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.957 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280205+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.957 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280456+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.957 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279651+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.957 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280405+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.957 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:32.641933+0000 30'100 43:135 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.958 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281745+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.958 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.282114+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.958 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279429+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.958 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279199+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.958 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279163+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.958 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.281429+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.958 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.279830+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.959 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280031+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.959 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280292+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.959 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:44:19.279863+0000 43'101 43:134 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.959 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:19.280584+0000 30'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:34.959 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:45:34.959 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:34.960 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:45:34.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:45:47.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:45:47.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:47.312 INFO:tasks.workunit.client.0.smithi142.stdout:killing divergent 1 2020-03-11T22:45:47.643 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:47.643 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.417340+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.643 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.413730+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.644 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410023+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.644 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.407779+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.644 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:45:33.411619+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.644 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410140+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.644 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410558+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.644 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.407340+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.644 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.407429+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.645 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410804+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.645 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.415827+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.645 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.410807+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.645 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.407001+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.645 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.407045+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.645 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:45:33.410578+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.646 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.416155+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.646 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.413248+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.646 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.406614+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.646 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.406479+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.646 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.405527+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.646 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.413997+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.646 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:45:33.416586+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.647 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.416920+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.647 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.405021+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.647 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.406209+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.647 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:45:33.408104+0000 30'100 48:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.647 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.414401+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.647 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.415285+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.647 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:45:33.405301+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.648 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:45:33.406765+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.648 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:45:33.406394+0000 43'101 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.648 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.413484+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:47.648 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:45:47.648 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:47.649 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:45:47.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:45:47.653 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:47.653 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:47.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:47.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:47.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:47.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:47.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17403.out 2020-03-11T22:45:47.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17403.out 2020-03-11T22:45:47.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:47.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:47.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:45:47.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:47.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:45:47.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17403.out 2020-03-11T22:45:48.467 INFO:tasks.workunit.client.0.smithi142.stderr:Exporting 1.0 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/48 n=3 ec=22/22 lis/c=47/47 les/c/f=48/48/0 sis=47 pruub=13.835257131s) 2020-03-11T22:45:48.480 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:45:48.505 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:45:48.525 INFO:tasks.workunit.client.0.smithi142.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:45:48.530 INFO:tasks.workunit.client.0.smithi142.stderr:Export successful 2020-03-11T22:45:48.530 INFO:tasks.workunit.client.0.smithi142.stdout: marking collection for removal 2020-03-11T22:45:48.530 INFO:tasks.workunit.client.0.smithi142.stdout:setting '_remove' omap key 2020-03-11T22:45:48.531 INFO:tasks.workunit.client.0.smithi142.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:45:48.554 INFO:tasks.workunit.client.0.smithi142.stderr:Remove successful 2020-03-11T22:45:49.047 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17403.out 2020-03-11T22:45:49.047 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:49.047 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:49.048 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:45:49.048 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:49.048 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:45:49.048 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17403.out 2020-03-11T22:45:49.510 INFO:tasks.workunit.client.0.smithi142.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={22=1,42=32}} deleted_pools ) 2020-03-11T22:45:49.510 INFO:tasks.workunit.client.0.smithi142.stdout:Importing pgid 1.0 2020-03-11T22:45:49.510 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:45:49.511 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:45:49.554 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:45:49.554 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:45:49.562 INFO:tasks.workunit.client.0.smithi142.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:45:49.563 INFO:tasks.workunit.client.0.smithi142.stdout:snapset 0=[]:{} 2020-03-11T22:45:49.571 INFO:tasks.workunit.client.0.smithi142.stdout:write_pg epoch 48 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/48 n=3 ec=22/22 lis/c=47/47 les/c/f=48/48/0 sis=47 pruub=13.835257131s) 2020-03-11T22:45:49.579 INFO:tasks.workunit.client.0.smithi142.stdout:Import successful 2020-03-11T22:45:50.098 INFO:tasks.workunit.client.0.smithi142.stdout:reviving divergent 1 2020-03-11T22:45:50.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:45:50.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:45:50.430 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:50.431 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.417340+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.431 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.413730+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.431 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410023+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.431 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.407779+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.431 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:45:33.411619+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.431 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410140+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.431 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410558+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.432 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.407340+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.432 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.407429+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.432 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.410804+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.432 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.415827+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.432 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.410807+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.432 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.407001+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.432 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.407045+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.433 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2020-03-11T22:45:33.410578+0000 30'100 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.433 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.416155+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.433 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.413248+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.433 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.406614+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.433 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.406479+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.433 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.405527+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.434 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.413997+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.434 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:45:33.416586+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.434 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.416920+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.434 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.405021+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.434 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.406209+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.434 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:45:33.408104+0000 30'100 48:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.434 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.414401+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.435 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:45:33.415285+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.435 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:45:33.405301+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.435 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:45:33.406765+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.435 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:45:33.406394+0000 43'101 48:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.435 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:45:33.413484+0000 30'100 48:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:45:50.435 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:45:50.436 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:50.436 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:45:50.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:45:50.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:50.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:50.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:45:50.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:50.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:45:50.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:50.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:50.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:50.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:50.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:45:50.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:45:50.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:50.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:50.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:50.445 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:50.446 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:50.446 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:50.446 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:50.446 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:50.446 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:45:50.446 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:45:50.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:50.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:50.448 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:45:50.448 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:45:50.448 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:50.448 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:45:50.448 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:45:50.449 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:45:50.449 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:50.449 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:50.450 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:50.469 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:50.467+0000 7f4709e8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:50.471 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:50.468+0000 7f4709e8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:50.473 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:50.470+0000 7f4709e8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:50.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:45:50.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:50.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:50.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:50.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:50.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:50.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:50.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:50.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:50.792 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:45:50.988 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:50.985+0000 7f4709e8dec0 -1 Falling back to public interface 2020-03-11T22:45:51.135 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [9,30) [v2:127.0.0.1:6818/42063,v1:127.0.0.1:6819/42063] [v2:127.0.0.1:6820/42063,v1:127.0.0.1:6821/42063] exists,up e79d27b4-6fc2-43ed-90f7-84fe096e508b 2020-03-11T22:45:51.135 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:45:51.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:51.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:51.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:51.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:45:51.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:51.136 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:45:51.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:51.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:51.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:51.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:51.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:51.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:45:51.479 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 32 last_clean_interval [9,30) [v2:127.0.0.1:6818/42063,v1:127.0.0.1:6819/42063] [v2:127.0.0.1:6820/42063,v1:127.0.0.1:6821/42063] exists,up e79d27b4-6fc2-43ed-90f7-84fe096e508b 2020-03-11T22:45:51.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:51.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:51.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:51.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:45:51.746 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:45:51.743+0000 7f4709e8dec0 -1 osd.1 48 log_to_monitors {default=true} 2020-03-11T22:46:11.481 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:46:11.481 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:46:11.482 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:46:11.482 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:11.482 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:11.482 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:11.482 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:46:11.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:46:11.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:46:11.579 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:46:11.579 INFO:tasks.workunit.client.0.smithi142.stdout: "ops": [], 2020-03-11T22:46:11.580 INFO:tasks.workunit.client.0.smithi142.stdout: "num_ops": 0 2020-03-11T22:46:11.580 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:46:11.587 INFO:tasks.workunit.client.0.smithi142.stdout:allowing recovery 2020-03-11T22:46:11.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:46:11.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:46:11.912 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:11.912 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:03.193397+0000 30'100 57:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.913 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:46:06.896662+0000 30'100 58:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:46:06.896614+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.913 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.206825+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.913 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:52.523159+0000 30'100 57:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.913 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:46:03.194382+0000 30'100 57:156 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.913 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:01.201737+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:45:55.724573+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.913 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.205547+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:45:53.719432+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.914 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:52.523319+0000 30'100 57:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.914 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:03.194902+0000 30'100 57:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.914 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:01.210962+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.914 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:46:03.596945+0000 30'100 60:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:46:03.596895+0000 30'100 2020-03-11T22:46:03.596895+0000 0 2020-03-11T22:46:11.914 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:46:01.731405+0000 30'100 57:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:46:01.731357+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.914 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:46:03.193463+0000 30'100 57:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.914 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:01.210064+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.915 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.206506+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.915 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:52.520907+0000 30'100 57:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.915 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:59.206042+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.915 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:46:00.691180+0000 30'100 57:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:46:00.691133+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.915 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:46:03.194665+0000 30'100 57:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.915 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:46:05.927986+0000 30'100 58:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:46:05.927933+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.915 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:52.521722+0000 30'100 57:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.916 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:59.208025+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.916 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:46:01.209866+0000 48'101 58:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.916 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:46:03.194040+0000 30'100 57:155 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:46:00.519720+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.916 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:45:59.660171+0000 30'100 57:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:59.660125+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.916 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:46:04.611810+0000 30'100 60:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:46:04.611762+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.916 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:46:01.210044+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.916 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:46:03.185329+0000 30'100 57:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.917 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:46:02.705944+0000 30'100 57:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:46:02.705899+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.917 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.207581+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.917 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:46:01.201815+0000 43'101 58:152 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.917 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:03.193645+0000 30'100 57:152 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:46:02.545483+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:11.917 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:46:11.917 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:11.918 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:46:11.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:46:11.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:46:12.043 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:12.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:46:12.051 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:46:12.171 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:12.178 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:46:12.179 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:46:12.299 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:12.307 INFO:tasks.workunit.client.0.smithi142.stdout:reading divergent objects 2020-03-11T22:46:12.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:46:12.307 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:46:12.636 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:12.636 INFO:tasks.workunit.client.0.smithi142.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:03.193397+0000 30'100 57:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.636 INFO:tasks.workunit.client.0.smithi142.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:46:06.896662+0000 30'100 58:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:46:06.896614+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.636 INFO:tasks.workunit.client.0.smithi142.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.206825+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.637 INFO:tasks.workunit.client.0.smithi142.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:52.523159+0000 30'100 57:147 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.637 INFO:tasks.workunit.client.0.smithi142.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:46:03.194382+0000 30'100 57:156 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.637 INFO:tasks.workunit.client.0.smithi142.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:01.201737+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:45:55.724573+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.637 INFO:tasks.workunit.client.0.smithi142.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.205547+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:45:53.719432+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.637 INFO:tasks.workunit.client.0.smithi142.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:52.523319+0000 30'100 57:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.637 INFO:tasks.workunit.client.0.smithi142.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:03.194902+0000 30'100 57:152 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.637 INFO:tasks.workunit.client.0.smithi142.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:01.210962+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.638 INFO:tasks.workunit.client.0.smithi142.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:46:03.596945+0000 30'100 60:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:46:03.596895+0000 30'100 2020-03-11T22:46:03.596895+0000 0 2020-03-11T22:46:12.638 INFO:tasks.workunit.client.0.smithi142.stdout:1.14 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:46:01.731405+0000 30'100 57:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:46:01.731357+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.638 INFO:tasks.workunit.client.0.smithi142.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:46:03.193463+0000 30'100 57:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.638 INFO:tasks.workunit.client.0.smithi142.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:01.210064+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.638 INFO:tasks.workunit.client.0.smithi142.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.206506+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.638 INFO:tasks.workunit.client.0.smithi142.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:52.520907+0000 30'100 57:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.638 INFO:tasks.workunit.client.0.smithi142.stdout:1.5 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:59.206042+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.639 INFO:tasks.workunit.client.0.smithi142.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:46:00.691180+0000 30'100 57:150 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:46:00.691133+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.639 INFO:tasks.workunit.client.0.smithi142.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:46:03.194665+0000 30'100 57:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.639 INFO:tasks.workunit.client.0.smithi142.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2020-03-11T22:46:05.927986+0000 30'100 58:154 [0,1,2] 0 [0,1,2] 0 30'100 2020-03-11T22:46:05.927933+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.639 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:45:52.521722+0000 30'100 57:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.639 INFO:tasks.workunit.client.0.smithi142.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:45:59.208025+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.639 INFO:tasks.workunit.client.0.smithi142.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:46:01.209866+0000 48'101 58:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.639 INFO:tasks.workunit.client.0.smithi142.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:46:03.194040+0000 30'100 57:155 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:46:00.519720+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.640 INFO:tasks.workunit.client.0.smithi142.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:45:59.660171+0000 30'100 57:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:45:59.660125+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.640 INFO:tasks.workunit.client.0.smithi142.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:46:04.611810+0000 30'100 60:136 [2,0,1] 2 [2,0,1] 2 30'100 2020-03-11T22:46:04.611762+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.640 INFO:tasks.workunit.client.0.smithi142.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:46:01.210044+0000 30'100 58:151 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.640 INFO:tasks.workunit.client.0.smithi142.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:46:03.185329+0000 30'100 57:154 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.640 INFO:tasks.workunit.client.0.smithi142.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:46:02.705944+0000 30'100 57:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-03-11T22:46:02.705899+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.640 INFO:tasks.workunit.client.0.smithi142.stdout:1.d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:45:59.207581+0000 30'100 60:133 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.640 INFO:tasks.workunit.client.0.smithi142.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:46:01.201815+0000 43'101 58:152 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:43:33.871767+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.641 INFO:tasks.workunit.client.0.smithi142.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:46:03.193645+0000 30'100 57:152 [1,2,0] 1 [1,2,0] 1 30'100 2020-03-11T22:46:02.545483+0000 0'0 2020-03-11T22:43:33.871767+0000 0 2020-03-11T22:46:12.641 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:46:12.641 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:12.641 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:46:12.646 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:46:12.647 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:46:12.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:12.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:46:12.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:12.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:46:12.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:12.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:46:12.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:12.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:46:12.864 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:46:12.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:46:12.865 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:46:12.913 INFO:tasks.workunit.client.0.smithi142.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:46:12.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:46:12.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:46:12.942 INFO:tasks.workunit.client.0.smithi142.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:46:12.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:46:12.944 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:46:12.945 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:46:12.953 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4542.0:1 2020-03-11T22:43:57.807418+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.953 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 20 _merge_object_divergent_entries: keeping 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4542.0:1 2020-03-11T22:43:57.807418+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.953 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:12.953 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-03-11T22:46:12.953 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:46:12.954 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:46:12.954 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 30'106 2020-03-11T22:46:12.954 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4542.0:1 2020-03-11T22:43:57.807418+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.954 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:46:12.954 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.141+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'89 2020-03-11T22:46:12.954 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 30'106 (30'4) delete 1:70b3bf12:::existing_4:head by client.4545.0:1 2020-03-11T22:43:57.811461+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:46:12.954 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 20 _merge_object_divergent_entries: keeping 30'106 (30'4) delete 1:70b3bf12:::existing_4:head by client.4545.0:1 2020-03-11T22:43:57.811461+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:46:12.955 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:12.955 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 30'106 last_divergent_update: 30'106 2020-03-11T22:46:12.955 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 10 _merge_object_divergent_entries: more recent entry found: 43'101 (30'4) modify 1:70b3bf12:::existing_4:head by client.4656.0:1 2020-03-11T22:44:51.722806+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:46:12.955 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 30'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4533.0:1 2020-03-11T22:43:57.791782+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.955 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 20 _merge_object_divergent_entries: keeping 30'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4533.0:1 2020-03-11T22:43:57.791782+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.955 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:12.955 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 30'101 last_divergent_update: 30'101 2020-03-11T22:46:12.956 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:46:12.956 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.142+0000 7f0909243700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:46:12.956 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4544.0:1 2020-03-11T22:43:57.809834+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:46:12.956 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 20 _merge_object_divergent_entries: keeping 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4544.0:1 2020-03-11T22:43:57.809834+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:46:12.956 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:46:12.956 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 30'105 last_divergent_update: 30'105 2020-03-11T22:46:12.956 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:46:12.957 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:46:12.957 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 30'106 2020-03-11T22:46:12.957 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 30'105 (30'3) delete 1:8881e6d8:::existing_3:head by client.4544.0:1 2020-03-11T22:43:57.809834+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:46:12.957 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:46:12.957 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.209+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'84 2020-03-11T22:46:12.957 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 30'103 (30'1) modify 1:d858c035:::existing_1:head by client.4540.0:1 2020-03-11T22:43:57.806718+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.957 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 20 _merge_object_divergent_entries: keeping 30'103 (30'1) modify 1:d858c035:::existing_1:head by client.4540.0:1 2020-03-11T22:43:57.806718+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.958 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:12.960 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-03-11T22:46:12.960 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:46:12.961 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:46:12.961 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 30'106 2020-03-11T22:46:12.961 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 30'103 (30'1) modify 1:d858c035:::existing_1:head by client.4540.0:1 2020-03-11T22:43:57.806718+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.961 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:46:12.961 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'90 2020-03-11T22:46:12.961 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 30'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4532.0:1 2020-03-11T22:43:57.791278+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.962 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 20 _merge_object_divergent_entries: keeping 30'102 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4532.0:1 2020-03-11T22:43:57.791278+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:12.962 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:12.962 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 30'102 last_divergent_update: 30'102 2020-03-11T22:46:12.962 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:46:12.962 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:45:33.210+0000 7f090a245700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:46:12.963 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:46:12.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:46:12.969 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:46:12.970 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:46:12.977 INFO:tasks.workunit.client.0.smithi142.stdout:success 2020-03-11T22:46:12.977 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:46:12.977 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17403.out 2020-03-11T22:46:12.978 INFO:tasks.workunit.client.0.smithi142.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17403.out': No such file or directory 2020-03-11T22:46:12.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:46:12.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:46:12.979 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:46:13.561 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' does not exist 2020-03-11T22:46:13.568 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:46:13.569 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:13.569 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:13.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:13.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:13.570 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:13.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:13.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:46:13.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:46:13.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:13.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:46:13.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:13.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:13.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:13.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:13.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:13.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:13.683 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:13.683 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:13.683 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:13.684 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:13.684 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:13.685 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:13.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:13.687 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:13.687 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:13.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:13.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:46:13.706 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:13.706 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.706 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:13.707 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:46:13.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:13.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:13.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:46:13.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:46:13.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:46:13.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:46:13.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:46:13.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:13.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:46:13.709 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:13.709 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:13.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:13.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:13.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:13.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:13.711 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:13.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:13.711 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:13.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:13.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:13.712 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:13.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:13.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:13.714 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:13.715 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:13.716 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:13.716 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:46:13.717 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:13.717 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.717 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:13.717 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:46:13.718 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:13.718 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:13.718 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:46:13.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:46:13.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:13.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:13.719 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17403 2020-03-11T22:46:13.721 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:46:13.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:46:13.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:46:13.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:46:13.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:46:13.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:46:13.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:46:13.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:46:13.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:46:13.723 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:46:13.723 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:46:13.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:46:13.724 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:46:13.724 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:46:13.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:46:13.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:46:13.724 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:46:13.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:46:13.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:46:13.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:46:13.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:46:14.004 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:46:14.004 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:14.005 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:14.005 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:14.005 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:14.005 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:14.005 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:14.005 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:46:14.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:46:14.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:46:14.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:14.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:14.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:46:14.173 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:46:14.174 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:14.174 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:14.174 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:14.174 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:14.174 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:14.175 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:14.175 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:46:14.175 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:14.175 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get fsid 2020-03-11T22:46:14.259 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:46:14.259 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:14.259 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:14.259 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:46:14.260 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:46:14.261 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:14.261 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:14.261 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:14.261 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:14.261 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:14.261 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:14.262 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-mon.a.asok 2020-03-11T22:46:14.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:14.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17403/ceph-mon.a.asok config get mon_host 2020-03-11T22:46:14.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:46:14.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:46:14.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:46:14.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:46:14.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:46:14.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:46:14.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:46:15.213 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:46:15.213 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:15.213 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:15.213 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:15.214 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:15.214 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:15.214 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:15.214 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:15.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:15.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:15.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:46:15.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:46:15.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:15.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:46:15.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:15.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:46:15.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:15.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:15.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:15.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:15.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:46:15.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:46:15.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:15.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:15.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:15.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:15.245 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:15.245 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:15.245 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:15.245 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:15.245 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:15.245 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:15.245 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:15.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:15.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:15.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:15.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:15.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:15.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:15.246 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:15.247 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:15.247 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:15.247 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:15.247 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:46:15.247 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:15.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=36e0b51c-dec2-42db-b07e-d9cf10df314e 2020-03-11T22:46:15.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 36e0b51c-dec2-42db-b07e-d9cf10df314e' 2020-03-11T22:46:15.248 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:15.248 INFO:tasks.workunit.client.0.smithi142.stdout:add osd0 36e0b51c-dec2-42db-b07e-d9cf10df314e 2020-03-11T22:46:15.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA3amley55mDxAAsRTwpgQhctpNoDqHXTC4yA== 2020-03-11T22:46:15.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA3amley55mDxAAsRTwpgQhctpNoDqHXTC4yA=="}' 2020-03-11T22:46:15.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 36e0b51c-dec2-42db-b07e-d9cf10df314e -i td/divergent-priors/0/new.json 2020-03-11T22:46:15.579 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:46:15.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:46:15.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA3amley55mDxAAsRTwpgQhctpNoDqHXTC4yA== --osd-uuid 36e0b51c-dec2-42db-b07e-d9cf10df314e 2020-03-11T22:46:15.617 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:15.613+0000 7f1e27179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:15.624 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:15.615+0000 7f1e27179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:15.624 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:15.617+0000 7f1e27179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:15.624 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:15.618+0000 7f1e27179ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:46:15.624 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:15.618+0000 7f1e27179ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:46:15.624 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:15.618+0000 7f1e27179ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:46:15.662 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:15.658+0000 7f1e27179ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:46:18.672 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:46:18.672 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:18.673 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd0 key to auth repository 2020-03-11T22:46:18.673 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:46:18.673 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:19.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:46:19.119 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:46:19.120 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:19.120 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:19.120 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:19.121 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:19.146 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:19.143+0000 7f991aa56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.148 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:19.145+0000 7f991aa56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.150 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:19.147+0000 7f991aa56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:46:19.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:19.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:19.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:19.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:19.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:19.467 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:19.467 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:19.467 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:19.467 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:46:19.664 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:19.661+0000 7f991aa56ec0 -1 Falling back to public interface 2020-03-11T22:46:19.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:20.370 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:20.368+0000 7f991aa56ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:46:20.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:20.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:20.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:20.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:20.814 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:46:20.815 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:21.193 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43560,v1:127.0.0.1:6803/43560] [v2:127.0.0.1:6804/43560,v1:127.0.0.1:6805/43560] exists,up 36e0b51c-dec2-42db-b07e-d9cf10df314e 2020-03-11T22:46:21.193 INFO:tasks.workunit.client.0.smithi142.stdout:add osd1 11c7e564-1ae2-4288-8725-ccd9a0814a84 2020-03-11T22:46:21.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:21.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:21.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:21.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:21.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:46:21.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:46:21.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:21.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:46:21.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:46:21.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:21.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:21.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:21.196 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:21.196 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:21.196 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:21.197 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:21.197 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:21.197 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:21.197 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:21.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:21.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:21.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:21.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:21.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:21.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:21.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:21.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:21.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:21.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:21.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:21.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:46:21.199 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:21.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=11c7e564-1ae2-4288-8725-ccd9a0814a84 2020-03-11T22:46:21.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 11c7e564-1ae2-4288-8725-ccd9a0814a84' 2020-03-11T22:46:21.199 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:21.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA9amle9BJyDBAAbWO5+0cdFBIU8LHUXKF//g== 2020-03-11T22:46:21.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA9amle9BJyDBAAbWO5+0cdFBIU8LHUXKF//g=="}' 2020-03-11T22:46:21.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 11c7e564-1ae2-4288-8725-ccd9a0814a84 -i td/divergent-priors/1/new.json 2020-03-11T22:46:21.657 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:46:21.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:46:21.667 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA9amle9BJyDBAAbWO5+0cdFBIU8LHUXKF//g== --osd-uuid 11c7e564-1ae2-4288-8725-ccd9a0814a84 2020-03-11T22:46:21.696 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:21.693+0000 7ff656c06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.702 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:21.699+0000 7ff656c06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.703 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:21.701+0000 7ff656c06ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:21.704 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:21.701+0000 7ff656c06ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:46:21.704 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:21.701+0000 7ff656c06ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:46:21.704 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:21.701+0000 7ff656c06ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:46:21.802 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:21.799+0000 7ff656c06ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:46:24.903 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:46:24.903 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:24.904 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd1 key to auth repository 2020-03-11T22:46:24.904 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:46:24.904 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:25.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:46:25.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:25.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:25.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:25.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:25.364 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:46:25.388 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:25.386+0000 7f435dc69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.390 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:25.387+0000 7f435dc69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.392 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:25.390+0000 7f435dc69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:25.707 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:46:25.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:25.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:25.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:25.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:25.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:25.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:25.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:25.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:25.709 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:46:25.907 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:25.904+0000 7f435dc69ec0 -1 Falling back to public interface 2020-03-11T22:46:26.056 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:26.700 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:26.697+0000 7f435dc69ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:46:27.058 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:27.059 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:27.059 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:27.059 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:27.059 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:27.059 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:46:27.408 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43915,v1:127.0.0.1:6811/43915] [v2:127.0.0.1:6812/43915,v1:127.0.0.1:6813/43915] exists,up 11c7e564-1ae2-4288-8725-ccd9a0814a84 2020-03-11T22:46:27.408 INFO:tasks.workunit.client.0.smithi142.stdout:add osd2 2c7e5aef-f703-4a53-8c31-d0d8c932abf4 2020-03-11T22:46:27.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:27.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:27.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:27.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:27.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:46:27.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:46:27.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:27.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:46:27.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:27.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:46:27.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:27.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:27.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:27.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:27.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:46:27.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:46:27.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:27.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:27.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:27.411 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:27.411 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:27.411 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:27.411 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:27.412 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:27.412 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:46:27.412 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:27.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:46:27.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:27.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:27.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:27.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:27.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:27.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:27.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:27.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:27.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:27.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:27.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:46:27.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:27.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2c7e5aef-f703-4a53-8c31-d0d8c932abf4 2020-03-11T22:46:27.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 2c7e5aef-f703-4a53-8c31-d0d8c932abf4' 2020-03-11T22:46:27.415 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:27.426 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBDamleeSY8GRAAzvk3ENxr2dXvNFFn8OsVuA== 2020-03-11T22:46:27.426 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBDamleeSY8GRAAzvk3ENxr2dXvNFFn8OsVuA=="}' 2020-03-11T22:46:27.426 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2c7e5aef-f703-4a53-8c31-d0d8c932abf4 -i td/divergent-priors/2/new.json 2020-03-11T22:46:27.887 INFO:tasks.workunit.client.0.smithi142.stdout:2 2020-03-11T22:46:27.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:46:27.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBDamleeSY8GRAAzvk3ENxr2dXvNFFn8OsVuA== --osd-uuid 2c7e5aef-f703-4a53-8c31-d0d8c932abf4 2020-03-11T22:46:27.926 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:27.923+0000 7f8b5de71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:27.928 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:27.925+0000 7f8b5de71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:27.930 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:27.927+0000 7f8b5de71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:27.931 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:27.928+0000 7f8b5de71ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:46:27.931 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:27.928+0000 7f8b5de71ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:46:27.931 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:27.928+0000 7f8b5de71ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:46:27.983 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:27.980+0000 7f8b5de71ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:46:30.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:46:30.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:30.876 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:46:30.877 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:30.877 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd2 key to auth repository 2020-03-11T22:46:31.334 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:46:31.334 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:31.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:31.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:31.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:31.335 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:46:31.361 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:31.359+0000 7fa5ea766ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:31.363 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:31.360+0000 7fa5ea766ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:31.365 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:31.362+0000 7fa5ea766ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:31.681 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:46:31.681 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:31.681 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:46:31.681 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:31.681 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:31.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:31.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:31.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:31.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:31.682 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:46:31.880 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:31.878+0000 7fa5ea766ec0 -1 Falling back to public interface 2020-03-11T22:46:32.060 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:32.632 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:46:32.629+0000 7fa5ea766ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:46:33.062 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:33.062 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:33.062 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:33.062 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:33.062 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:46:33.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:33.417 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44270,v1:127.0.0.1:6819/44270] [v2:127.0.0.1:6820/44270,v1:127.0.0.1:6821/44270] exists,up 2c7e5aef-f703-4a53-8c31-d0d8c932abf4 2020-03-11T22:46:33.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:33.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:33.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:33.418 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:46:34.026 INFO:tasks.workunit.client.0.smithi142.stderr:noout is set 2020-03-11T22:46:34.033 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:46:34.711 INFO:tasks.workunit.client.0.smithi142.stderr:noin is set 2020-03-11T22:46:34.718 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:46:35.378 INFO:tasks.workunit.client.0.smithi142.stderr:nodown is set 2020-03-11T22:46:35.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:46:35.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:46:35.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:46:35.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:46:35.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:46:35.388 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:46:35.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:46:35.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:46:36.582 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' already exists 2020-03-11T22:46:36.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:46:37.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:46:37.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:46:38.259 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:46:38.267 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:46:38.267 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:38.267 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:38.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:38.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:38.268 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:38.269 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:38.269 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:38.269 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:38.269 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:38.269 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:38.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:38.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:38.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:38.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:38.361 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:38.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:38.725 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:46:38.725 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:46:38.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:38.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:38.726 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:38.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:46:38.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:46:38.855 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:38.855 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:38.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:46:38.985 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:46:38.985 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:38.985 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:39.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:46:39.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509444' 2020-03-11T22:46:39.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:39.116 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:39.116 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:39.117 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:39.117 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:46:39.117 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:39.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:46:39.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:46:39.119 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:39.119 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:46:39.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:46:39.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:40.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:40.463 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:40.815 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:46:40.815 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:40.816 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:46:40.816 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:40.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:40.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:46:40.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:40.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:46:40.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:46:40.818 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:46:40.819 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:41.158 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:46:41.158 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:41.159 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:46:41.159 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:41.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:41.160 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:46:41.160 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:41.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:46:41.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:46:41.162 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:41.162 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:46:41.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:46:41.504 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:41.504 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:41.504 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:41.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:46:41.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:41.947 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:41.947 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:41.947 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:41.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:41.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:41.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:42.277 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:46:42.278 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:42.278 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:42.278 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:42.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:46:42.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:42.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:42.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:46:42.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:46:42.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:42.728 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:43.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:43.070 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:46:43.070 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:46:43.070 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:43.070 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:43.070 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:43.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:46:43.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:46:43.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:43.201 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:43.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:46:43.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2020-03-11T22:46:43.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:43.330 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:43.458 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:46:43.459 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-64424509446' 2020-03-11T22:46:43.459 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:43.459 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:46:43.459 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:43.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:43.460 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:46:43.461 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:43.461 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:46:43.462 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:46:43.462 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:43.462 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:46:43.801 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:46:43.801 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:44.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:44.803 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:45.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:46:45.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:45.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:46:45.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:45.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:45.164 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:46:45.164 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:45.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:46:45.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:46:45.165 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:46:45.165 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:45.511 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672969 2020-03-11T22:46:45.511 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:45.512 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:46:45.512 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:45.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:45.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:46:45.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:45.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:46:45.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:46:45.515 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:45.515 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:46:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509447 -lt 64424509446 2020-03-11T22:46:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:46:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:45.874 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:46:45.874 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:45.874 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:45.875 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:45.875 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:45.875 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:45.966 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:45.966 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:45.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:45.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:45.967 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:46.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:46.312 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:46:46.312 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:46:46.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:46.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:46.312 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:46.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836491 2020-03-11T22:46:46.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491' 2020-03-11T22:46:46.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:46.441 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:46.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672971 2020-03-11T22:46:46.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672971' 2020-03-11T22:46:46.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:46.580 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:46.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-03-11T22:46:46.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509449' 2020-03-11T22:46:46.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:46.710 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836491 2020-03-11T22:46:46.711 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:46.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:46.712 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836491 2020-03-11T22:46:46.712 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:46.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836491 2020-03-11T22:46:46.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-03-11T22:46:46.713 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:46.713 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836491 2020-03-11T22:46:47.053 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836491 2020-03-11T22:46:47.053 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:47.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672971 2020-03-11T22:46:47.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:47.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:47.055 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672971 2020-03-11T22:46:47.055 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:47.056 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672971 2020-03-11T22:46:47.056 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2020-03-11T22:46:47.056 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672971 2020-03-11T22:46:47.056 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:47.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672971 2020-03-11T22:46:47.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:48.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:48.424 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:48.799 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672971 2020-03-11T22:46:48.799 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:48.800 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509449 2020-03-11T22:46:48.800 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:48.800 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:46:48.801 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509449 2020-03-11T22:46:48.801 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:48.801 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509449 2020-03-11T22:46:48.802 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-03-11T22:46:48.802 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:46:48.802 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509449 2020-03-11T22:46:49.144 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509449 -lt 64424509449 2020-03-11T22:46:49.145 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:46:49.145 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:49.145 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:49.594 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:46:49.594 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:46:49.595 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:46:49.595 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:46:49.595 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:46:49.595 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:46:49.595 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:46:49.595 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:46:49.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:46:49.923 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:46:49.923 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:46:49.923 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:46:50.366 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:46:50.366 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:46:50.367 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:46:50.367 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:46:50.367 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:46:50.688 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:46:50.696 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:46:50.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:46:50.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:50.697 INFO:tasks.workunit.client.0.smithi142.stdout:primary and soon to be divergent is 1 2020-03-11T22:46:51.017 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:51.017 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:46:36.690000+0000 0'0 27:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:46:51.017 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:46:51.017 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:51.018 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:46:51.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:46:51.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:51.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:46:51.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:46:51.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:51.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:46:51.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:46:51.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:51.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:46:51.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:46:51.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:46:51.029 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:46:51.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.030 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:46:51.030 INFO:tasks.workunit.client.0.smithi142.stdout:writing initial objects 2020-03-11T22:46:51.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:46:51.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:46:51.158 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.158 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:46:51.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:46:51.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:46:51.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:46:51.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:46:51.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:46:51.482 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:46:51.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:46:51.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:46:51.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:46:51.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:46:51.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:46:51.825 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.825 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:46:51.866 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.867 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:46:51.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:51.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:46:52.025 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.026 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:46:52.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:46:52.151 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.151 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:46:52.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:46:52.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:46:52.275 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.275 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:46:52.317 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.317 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:46:52.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:46:52.458 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.459 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:46:52.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.500 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:46:52.619 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.619 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:46:52.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.667 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:46:52.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.769 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:46:52.817 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.817 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:46:52.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:46:52.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:46:52.986 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:52.986 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:46:53.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:46:53.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:46:53.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:46:53.266 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.267 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:46:53.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:46:53.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:46:53.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:46:53.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:46:53.525 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.525 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:46:53.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:46:53.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:46:53.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:46:53.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:46:53.850 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.850 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:46:53.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:46:53.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:53.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:46:54.025 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.025 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:46:54.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:46:54.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:46:54.242 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:46:54.284 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.284 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:46:54.327 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.327 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:46:54.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:46:54.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.482 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:46:54.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:46:54.617 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.617 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:46:54.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:46:54.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:46:54.815 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:46:54.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:54.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:46:55.009 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.009 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:46:55.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:46:55.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:46:55.216 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.216 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:46:55.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:46:55.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.393 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:46:55.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:46:55.476 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.476 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:46:55.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:46:55.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:46:55.642 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.643 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:46:55.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:46:55.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:46:55.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:46:55.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:46:55.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:46:55.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:55.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:46:56.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:46:56.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:46:56.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:46:56.209 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:46:56.328 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:46:56.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.468 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:46:56.509 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.510 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:46:56.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:46:56.634 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.635 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:46:56.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:46:56.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:46:56.810 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.810 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:46:56.895 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.896 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:46:56.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:56.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:46:57.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:57.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:46:57.076 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:57.077 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:46:57.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:46:57.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:46:57.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:46:57.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:46:57.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:46:57.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:46:57.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:46:57.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:46:57.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:46:57.187 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:46:57.187 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:46:57.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:46:57.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:46:57.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:46:57.243 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:46:57.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:46:57.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:46:57.244 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:46:57.244 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:46:57.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:46:57.586 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:46:57.587 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:46:57.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:46:57.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:57.587 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:46:57.715 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836496 2020-03-11T22:46:57.716 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496' 2020-03-11T22:46:57.716 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:57.716 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:46:57.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672975 2020-03-11T22:46:57.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672975' 2020-03-11T22:46:57.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:46:57.845 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:46:57.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509453 2020-03-11T22:46:57.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509453' 2020-03-11T22:46:57.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:57.975 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836496 2020-03-11T22:46:57.975 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:57.977 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:46:57.977 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836496 2020-03-11T22:46:57.977 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:57.977 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836496 2020-03-11T22:46:57.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-03-11T22:46:57.978 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:57.978 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836496 2020-03-11T22:46:58.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836496 2020-03-11T22:46:58.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:46:59.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:46:59.323 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:46:59.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836496 -lt 21474836496 2020-03-11T22:46:59.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:46:59.726 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672975 2020-03-11T22:46:59.726 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:46:59.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:46:59.727 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672975 2020-03-11T22:46:59.728 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:46:59.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672975 2020-03-11T22:46:59.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2020-03-11T22:46:59.729 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:46:59.729 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672975 2020-03-11T22:47:00.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 42949672975 2020-03-11T22:47:00.075 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:00.076 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509453 2020-03-11T22:47:00.076 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:00.077 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:47:00.077 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509453 2020-03-11T22:47:00.077 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:00.078 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509453 2020-03-11T22:47:00.078 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2020-03-11T22:47:00.079 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509453 2020-03-11T22:47:00.079 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:47:00.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509453 -lt 64424509453 2020-03-11T22:47:00.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:00.424 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:00.424 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:00.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:47:00.871 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:00.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:00.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:00.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:00.871 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:00.872 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:00.872 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:01.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:47:01.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:01.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:01.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:01.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:47:01.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:47:01.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:47:01.649 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:47:01.649 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:47:01.649 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:47:01.649 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:47:01.649 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:47:01.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:47:01.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:47:01.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:47:01.997 INFO:tasks.workunit.client.0.smithi142.stdout:blackholing osds 0 2 2020-03-11T22:47:02.316 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:02.316 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:46:36.690000+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:47:02.316 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:47:02.316 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:02.317 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:47:02.325 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:02.326 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:47:02.326 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:47:02.326 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:47:02.326 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:02.327 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:02.327 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:47:02.327 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:47:02.327 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:47:02.328 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:47:02.412 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:47:02.412 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:47:02.412 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:47:02.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:02.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:47:02.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:47:02.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:47:02.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:02.420 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:02.421 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:47:02.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:47:02.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:47:02.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:47:02.507 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:47:02.507 INFO:tasks.workunit.client.0.smithi142.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:47:02.508 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:47:02.514 INFO:tasks.workunit.client.0.smithi142.stdout:writing divergent object 2020-03-11T22:47:02.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:47:02.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:47:02.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:47:03.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:47:03.516 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:47:03.674 INFO:tasks.workunit.client.0.smithi142.stdout:created pool test snap snap1 2020-03-11T22:47:03.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:47:03.676 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:47:03.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:47:13.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:47:13.682 INFO:tasks.workunit.client.0.smithi142.stderr:rados: no process found 2020-03-11T22:47:13.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:47:13.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:47:13.682 INFO:tasks.workunit.client.0.smithi142.stdout:killing all the osds 2020-03-11T22:47:14.012 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:14.013 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:46:36.690000+0000 29'104 29:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:47:14.013 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:47:14.013 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:14.014 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:47:14.022 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:47:14.022 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:14.022 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:14.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:14.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:14.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:14.129 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:14.129 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:47:14.129 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:47:14.629 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already down. 2020-03-11T22:47:14.637 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:47:14.637 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:47:15.137 INFO:tasks.workunit.client.0.smithi142.stderr:osd.1 is already down. 2020-03-11T22:47:15.145 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:47:15.145 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:47:15.705 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already down. 2020-03-11T22:47:15.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:15.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:47:16.213 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already out. 2020-03-11T22:47:16.221 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:16.222 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:47:16.764 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already out. 2020-03-11T22:47:16.772 INFO:tasks.workunit.client.0.smithi142.stdout:bringing up non_divergent 0 2 2020-03-11T22:47:16.772 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:47:16.773 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:47:17.093 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:17.093 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:46:36.690000+0000 29'104 29:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:47:17.093 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:47:17.093 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:17.094 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:47:17.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:17.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:47:17.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:47:17.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:17.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:47:17.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:17.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:47:17.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:47:17.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:17.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:17.108 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:17.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:47:17.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:47:17.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:17.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:17.109 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:47:17.110 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:17.110 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:17.110 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:17.110 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:17.110 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:17.110 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:47:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:47:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:47:17.113 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:17.113 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:47:17.113 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:47:17.113 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:17.113 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:17.114 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:17.114 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:47:17.132 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:17.129+0000 7f6b24a02ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:17.133 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:17.131+0000 7f6b24a02ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:17.135 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:17.132+0000 7f6b24a02ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:17.455 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:47:17.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:17.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:47:17.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:17.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:17.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:17.456 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:17.457 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:17.457 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:17.457 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:47:17.652 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:17.649+0000 7f6b24a02ec0 -1 Falling back to public interface 2020-03-11T22:47:17.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:18.467 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:18.464+0000 7f6b24a02ec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:47:18.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:18.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:18.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:18.807 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:47:18.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:18.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:19.159 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/48673,v1:127.0.0.1:6803/48673] [v2:127.0.0.1:6804/48673,v1:127.0.0.1:6805/48673] exists,up 36e0b51c-dec2-42db-b07e-d9cf10df314e 2020-03-11T22:47:19.159 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:47:19.159 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:47:19.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:19.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:47:19.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:47:19.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:19.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:19.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:19.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:19.162 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:47:19.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:19.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:19.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:19.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:19.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:19.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:47:19.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:47:19.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:19.165 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:47:19.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:47:19.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:19.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:19.165 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:19.184 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:19.181+0000 7f8462b85ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:19.186 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:19.183+0000 7f8462b85ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:19.188 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:19.185+0000 7f8462b85ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:19.513 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:47:19.513 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:19.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:47:19.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:19.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:19.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:19.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:19.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:19.515 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:19.515 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:47:19.703 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:19.700+0000 7f8462b85ec0 -1 Falling back to public interface 2020-03-11T22:47:19.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:20.646 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:47:20.643+0000 7f8462b85ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:47:20.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:20.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:20.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:20.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:20.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:20.859 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:47:21.202 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/48867,v1:127.0.0.1:6811/48867] [v2:127.0.0.1:6812/48867,v1:127.0.0.1:6813/48867] exists,up 2c7e5aef-f703-4a53-8c31-d0d8c932abf4 2020-03-11T22:47:21.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:21.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:21.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:21.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:21.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:47:21.705 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already in. 2020-03-11T22:47:21.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:47:21.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:47:22.337 INFO:tasks.workunit.client.0.smithi142.stderr:osd.2 is already in. 2020-03-11T22:47:22.345 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:47:27.347 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:47:27.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:47:27.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:47:27.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:47:27.349 INFO:tasks.workunit.client.0.smithi142.stdout:writing non-divergent object existing_4 2020-03-11T22:47:27.679 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:27.679 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:47:22.463434+0000 28'100 39:122 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:47:27.679 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:47:27.679 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:27.681 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:47:27.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:47:27.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:47:27.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:47:27.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:27.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:27.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:27.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:27.735 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:47:27.735 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:27.735 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:27.735 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:27.735 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:27.735 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:27.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:27.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:27.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:27.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:27.792 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:28.134 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:28.134 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:47:28.135 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:47:28.135 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:28.135 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:28.135 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:28.263 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855364 2020-03-11T22:47:28.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855364' 2020-03-11T22:47:28.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:28.264 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:28.355 INFO:tasks.workunit.client.0.smithi142.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:47:28.362 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:47:28.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855364 1-' 2020-03-11T22:47:28.363 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:28.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:47:28.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822660 2020-03-11T22:47:28.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855364 1- 2-154618822660' 2020-03-11T22:47:28.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:28.494 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855364 2020-03-11T22:47:28.495 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:28.495 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:28.496 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855364 2020-03-11T22:47:28.496 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:28.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855364 2020-03-11T22:47:28.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2020-03-11T22:47:28.497 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 150323855364 2020-03-11T22:47:28.497 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:28.838 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855364 2020-03-11T22:47:28.838 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:28.839 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:47:28.839 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:28.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:28.840 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:47:28.840 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:28.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:47:28.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:47:28.842 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:28.842 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 2020-03-11T22:47:29.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672979 -lt 2020-03-11T22:47:29.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672979: unary operator expected 2020-03-11T22:47:29.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:29.191 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822660 2020-03-11T22:47:29.192 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:29.192 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:47:29.192 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822660 2020-03-11T22:47:29.193 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:29.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822660 2020-03-11T22:47:29.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822660' 2020-03-11T22:47:29.194 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 154618822660 2020-03-11T22:47:29.194 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:47:29.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822658 -lt 154618822660 2020-03-11T22:47:29.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:30.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:30.535 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:47:30.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822660 -lt 154618822660 2020-03-11T22:47:30.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:30.888 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:30.889 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:31.332 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:47:31.332 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:31.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:31.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:31.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:31.332 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:31.333 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:31.333 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:31.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:31.661 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:31.661 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:31.661 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:32.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:32.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:47:32.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:32.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:47:32.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:32.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:32.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:32.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:32.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:32.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:32.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:32.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:32.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:32.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:32.531 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:32.532 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:32.532 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:32.532 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:32.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:32.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:32.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:32.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:32.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:32.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:32.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:32.976 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:32.976 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:33.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:33.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:33.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:47:33.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:33.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:47:33.620 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:33.620 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:33.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:33.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:33.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:33.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:33.622 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:33.622 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:33.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:33.949 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:33.950 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:33.950 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:34.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:34.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:34.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:34.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:34.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:34.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:34.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:34.398 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:34.398 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:34.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:34.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:34.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:47:34.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:34.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:47:35.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:35.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:35.264 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:35.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:35.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:35.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:35.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:35.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:35.597 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:35.597 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:35.597 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:35.598 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:36.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:36.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:36.040 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:36.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:36.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:36.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:36.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:36.041 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:36.041 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:36.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:36.480 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:36.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:47:36.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:36.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:47:37.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:37.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:37.282 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:37.282 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:37.283 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:37.283 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:37.283 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:37.283 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:37.620 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:37.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:37.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:37.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:38.060 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:38.060 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:38.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:38.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:38.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:38.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:38.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:38.061 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:38.061 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:38.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:38.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:38.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:47:38.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:38.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:47:40.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:40.107 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:40.107 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:40.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:40.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:40.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:40.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:40.108 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:40.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:40.445 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:40.445 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:40.446 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:40.904 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:40.904 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:40.904 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:40.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:40.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:40.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:40.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:40.905 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:40.905 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:41.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:41.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:41.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:47:41.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:41.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:47:44.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:44.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:44.560 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:44.561 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:44.561 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:44.561 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:44.561 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:44.561 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:44.901 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:44.902 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:44.902 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:44.902 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:45.347 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:45.347 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:45.347 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:45.347 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:45.347 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:45.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:45.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:45.348 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:45.348 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:45.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:45.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:45.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:47:45.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:45.788 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:47:52.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:52.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:52.189 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:52.190 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:52.190 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:52.190 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:52.190 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:52.190 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:52.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:52.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:52.529 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:52.529 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:52.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:52.972 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:47:52.972 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:47:52.972 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:47:52.972 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:47:52.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:47:52.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:47:52.973 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:47:52.973 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:47:53.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:47:53.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:47:53.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:47:53.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:53.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:48:00.717 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:00.718 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:00.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:00.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:00.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:00.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:00.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:00.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:01.060 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:01.061 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:01.061 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:01.062 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:01.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:01.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:01.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:01.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:01.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:01.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:01.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:01.507 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:01.507 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:01.946 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:01.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:01.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:48:01.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:48:02.382 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:48:02.382 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_fingerprint": "c22feccd-ec9a-415b-bb61-741d5acf9558", 2020-03-11T22:48:02.383 INFO:tasks.workunit.client.0.smithi142.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:48:02.383 INFO:tasks.workunit.client.0.smithi142.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:48:02.383 INFO:tasks.workunit.client.0.smithi142.stdout: "timestamp": "2020-03-11T22:48:02.378419+0000", 2020-03-11T22:48:02.383 INFO:tasks.workunit.client.0.smithi142.stdout: "tag": "", 2020-03-11T22:48:02.383 INFO:tasks.workunit.client.0.smithi142.stdout: "health": { 2020-03-11T22:48:02.383 INFO:tasks.workunit.client.0.smithi142.stdout: "status": "HEALTH_WARN", 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: "checks": { 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 17 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.384 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [], 2020-03-11T22:48:02.385 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:02.385 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.385 INFO:tasks.workunit.client.0.smithi142.stdout: "OSD_DOWN": { 2020-03-11T22:48:02.385 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:02.385 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:02.385 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 osds down", 2020-03-11T22:48:02.385 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:48:02.386 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.386 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:48:02.386 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.386 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "osd.1 (root=default,host=smithi142) is down" 2020-03-11T22:48:02.386 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.386 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.386 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:02.387 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.387 INFO:tasks.workunit.client.0.smithi142.stdout: "PG_DEGRADED": { 2020-03-11T22:48:02.387 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:02.387 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:02.387 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:48:02.387 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:48:02.387 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.388 INFO:tasks.workunit.client.0.smithi142.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:48:02.389 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:02.389 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:02.389 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:48:02.389 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:48:02.389 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.390 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:48:02.390 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.390 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:48:02.390 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.390 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.391 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:48:02.391 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.391 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.391 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:02.391 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.391 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.391 INFO:tasks.workunit.client.0.smithi142.stdout: "mutes": [] 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_first_committed": 1, 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_last_committed": 1, 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap": { 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:46:13.742457Z", 2020-03-11T22:48:02.392 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:46:13.742457Z", 2020-03-11T22:48:02.393 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release": 15, 2020-03-11T22:48:02.393 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:48:02.393 INFO:tasks.workunit.client.0.smithi142.stdout: "features": { 2020-03-11T22:48:02.393 INFO:tasks.workunit.client.0.smithi142.stdout: "persistent": [ 2020-03-11T22:48:02.394 INFO:tasks.workunit.client.0.smithi142.stdout: "kraken", 2020-03-11T22:48:02.394 INFO:tasks.workunit.client.0.smithi142.stdout: "luminous", 2020-03-11T22:48:02.394 INFO:tasks.workunit.client.0.smithi142.stdout: "mimic", 2020-03-11T22:48:02.394 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap-prune", 2020-03-11T22:48:02.395 INFO:tasks.workunit.client.0.smithi142.stdout: "nautilus", 2020-03-11T22:48:02.395 INFO:tasks.workunit.client.0.smithi142.stdout: "octopus" 2020-03-11T22:48:02.395 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.395 INFO:tasks.workunit.client.0.smithi142.stdout: "optional": [] 2020-03-11T22:48:02.396 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.396 INFO:tasks.workunit.client.0.smithi142.stdout: "mons": [ 2020-03-11T22:48:02.396 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.396 INFO:tasks.workunit.client.0.smithi142.stdout: "rank": 0, 2020-03-11T22:48:02.396 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "a", 2020-03-11T22:48:02.397 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:02.397 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.397 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.397 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 0 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.398 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.399 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.399 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:48:02.399 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:48:02.399 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 0, 2020-03-11T22:48:02.400 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 0 2020-03-11T22:48:02.400 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.400 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.400 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.401 INFO:tasks.workunit.client.0.smithi142.stdout: "quorum": [ 2020-03-11T22:48:02.401 INFO:tasks.workunit.client.0.smithi142.stdout: 0 2020-03-11T22:48:02.401 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.401 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap": { 2020-03-11T22:48:02.402 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 39, 2020-03-11T22:48:02.402 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:48:02.402 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:46:14.433398+0000", 2020-03-11T22:48:02.402 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:47:22.207432+0000", 2020-03-11T22:48:02.403 INFO:tasks.workunit.client.0.smithi142.stdout: "last_up_change": "2020-03-11T22:47:20.705031+0000", 2020-03-11T22:48:02.403 INFO:tasks.workunit.client.0.smithi142.stdout: "last_in_change": "2020-03-11T22:47:21.998697+0000", 2020-03-11T22:48:02.403 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:48:02.403 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_num": 5800384, 2020-03-11T22:48:02.403 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_set": [ 2020-03-11T22:48:02.404 INFO:tasks.workunit.client.0.smithi142.stdout: "nodown", 2020-03-11T22:48:02.404 INFO:tasks.workunit.client.0.smithi142.stdout: "noin", 2020-03-11T22:48:02.404 INFO:tasks.workunit.client.0.smithi142.stdout: "noout", 2020-03-11T22:48:02.404 INFO:tasks.workunit.client.0.smithi142.stdout: "pglog_hardlimit", 2020-03-11T22:48:02.404 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snapdirs", 2020-03-11T22:48:02.405 INFO:tasks.workunit.client.0.smithi142.stdout: "recovery_deletes", 2020-03-11T22:48:02.405 INFO:tasks.workunit.client.0.smithi142.stdout: "sortbitwise" 2020-03-11T22:48:02.405 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.405 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_version": 8, 2020-03-11T22:48:02.406 INFO:tasks.workunit.client.0.smithi142.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:48:02.406 INFO:tasks.workunit.client.0.smithi142.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:48:02.406 INFO:tasks.workunit.client.0.smithi142.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:48:02.406 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_snapshot": "", 2020-03-11T22:48:02.406 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_max": 1, 2020-03-11T22:48:02.407 INFO:tasks.workunit.client.0.smithi142.stdout: "max_osd": 3, 2020-03-11T22:48:02.409 INFO:tasks.workunit.client.0.smithi142.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:48:02.409 INFO:tasks.workunit.client.0.smithi142.stdout: "min_compat_client": "jewel", 2020-03-11T22:48:02.409 INFO:tasks.workunit.client.0.smithi142.stdout: "require_osd_release": "octopus", 2020-03-11T22:48:02.409 INFO:tasks.workunit.client.0.smithi142.stdout: "pools": [ 2020-03-11T22:48:02.409 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.410 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:48:02.410 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_name": "test", 2020-03-11T22:48:02.410 INFO:tasks.workunit.client.0.smithi142.stdout: "create_time": "2020-03-11T22:46:36.237276+0000", 2020-03-11T22:48:02.410 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": 16389, 2020-03-11T22:48:02.410 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:48:02.411 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 3, 2020-03-11T22:48:02.411 INFO:tasks.workunit.client.0.smithi142.stdout: "size": 3, 2020-03-11T22:48:02.411 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 2, 2020-03-11T22:48:02.411 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_rule": 1, 2020-03-11T22:48:02.412 INFO:tasks.workunit.client.0.smithi142.stdout: "object_hash": 2, 2020-03-11T22:48:02.412 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:48:02.412 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num": 1, 2020-03-11T22:48:02.412 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num": 1, 2020-03-11T22:48:02.412 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num_target": 1, 2020-03-11T22:48:02.413 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_target": 1, 2020-03-11T22:48:02.413 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_pending": 1, 2020-03-11T22:48:02.413 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pg_merge_meta": { 2020-03-11T22:48:02.413 INFO:tasks.workunit.client.0.smithi142.stdout: "source_pgid": "0.0", 2020-03-11T22:48:02.413 INFO:tasks.workunit.client.0.smithi142.stdout: "ready_epoch": 0, 2020-03-11T22:48:02.414 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_started": 0, 2020-03-11T22:48:02.414 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_clean": 0, 2020-03-11T22:48:02.414 INFO:tasks.workunit.client.0.smithi142.stdout: "source_version": "0'0", 2020-03-11T22:48:02.414 INFO:tasks.workunit.client.0.smithi142.stdout: "target_version": "0'0" 2020-03-11T22:48:02.415 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.415 INFO:tasks.workunit.client.0.smithi142.stdout: "last_change": "29", 2020-03-11T22:48:02.415 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend": "0", 2020-03-11T22:48:02.415 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:48:02.416 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:48:02.416 INFO:tasks.workunit.client.0.smithi142.stdout: "auid": 0, 2020-03-11T22:48:02.416 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_mode": "pool", 2020-03-11T22:48:02.416 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_seq": 1, 2020-03-11T22:48:02.416 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_epoch": 29, 2020-03-11T22:48:02.417 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_snaps": [ 2020-03-11T22:48:02.417 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.417 INFO:tasks.workunit.client.0.smithi142.stdout: "snapid": 1, 2020-03-11T22:48:02.417 INFO:tasks.workunit.client.0.smithi142.stdout: "stamp": "2020-03-11T22:47:03.541351+0000", 2020-03-11T22:48:02.418 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "snap1" 2020-03-11T22:48:02.418 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.418 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.418 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps": "[]", 2020-03-11T22:48:02.419 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_bytes": 0, 2020-03-11T22:48:02.419 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_objects": 0, 2020-03-11T22:48:02.419 INFO:tasks.workunit.client.0.smithi142.stdout: "tiers": [], 2020-03-11T22:48:02.419 INFO:tasks.workunit.client.0.smithi142.stdout: "tier_of": -1, 2020-03-11T22:48:02.420 INFO:tasks.workunit.client.0.smithi142.stdout: "read_tier": -1, 2020-03-11T22:48:02.420 INFO:tasks.workunit.client.0.smithi142.stdout: "write_tier": -1, 2020-03-11T22:48:02.420 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_mode": "none", 2020-03-11T22:48:02.420 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_bytes": 0, 2020-03-11T22:48:02.420 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_objects": 0, 2020-03-11T22:48:02.421 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:48:02.421 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:48:02.421 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:48:02.422 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_flush_age": 0, 2020-03-11T22:48:02.422 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_evict_age": 0, 2020-03-11T22:48:02.422 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:48:02.422 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_params": { 2020-03-11T22:48:02.422 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "none" 2020-03-11T22:48:02.424 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.424 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_period": 0, 2020-03-11T22:48:02.424 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_count": 0, 2020-03-11T22:48:02.424 INFO:tasks.workunit.client.0.smithi142.stdout: "use_gmt_hitset": true, 2020-03-11T22:48:02.424 INFO:tasks.workunit.client.0.smithi142.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:48:02.425 INFO:tasks.workunit.client.0.smithi142.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:48:02.425 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:48:02.425 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:48:02.425 INFO:tasks.workunit.client.0.smithi142.stdout: "grade_table": [], 2020-03-11T22:48:02.426 INFO:tasks.workunit.client.0.smithi142.stdout: "stripe_width": 8192, 2020-03-11T22:48:02.426 INFO:tasks.workunit.client.0.smithi142.stdout: "expected_num_objects": 0, 2020-03-11T22:48:02.426 INFO:tasks.workunit.client.0.smithi142.stdout: "fast_read": false, 2020-03-11T22:48:02.426 INFO:tasks.workunit.client.0.smithi142.stdout: "options": {}, 2020-03-11T22:48:02.426 INFO:tasks.workunit.client.0.smithi142.stdout: "application_metadata": {} 2020-03-11T22:48:02.427 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.427 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.427 INFO:tasks.workunit.client.0.smithi142.stdout: "osds": [ 2020-03-11T22:48:02.427 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.427 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:02.428 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "36e0b51c-dec2-42db-b07e-d9cf10df314e", 2020-03-11T22:48:02.428 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:48:02.428 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:48:02.428 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:48:02.428 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:48:02.429 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 5, 2020-03-11T22:48:02.429 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:48:02.429 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 35, 2020-03-11T22:48:02.429 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 38, 2020-03-11T22:48:02.429 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 30, 2020-03-11T22:48:02.430 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:48:02.430 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:02.430 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.430 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.430 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.431 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:48:02.431 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.431 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.431 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.431 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.432 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:48:02.432 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.432 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.432 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.433 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.433 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:48:02.433 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.433 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.434 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.434 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:48:02.434 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.434 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.434 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.435 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.435 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:48:02.435 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.435 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.435 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.436 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.436 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:48:02.436 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.436 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.436 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.437 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:48:02.437 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.437 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.437 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.438 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.438 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:48:02.438 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.438 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.438 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.439 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.439 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:48:02.439 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.439 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.439 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.440 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.441 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:48:02.441 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48673 2020-03-11T22:48:02.441 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.442 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.442 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.442 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6803/48673", 2020-03-11T22:48:02.442 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6805/48673", 2020-03-11T22:48:02.442 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6809/48673", 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6807/48673", 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:48:02.443 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:48:02.444 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.444 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.444 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.444 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:02.445 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "11c7e564-1ae2-4288-8725-ccd9a0814a84", 2020-03-11T22:48:02.445 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 0, 2020-03-11T22:48:02.445 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:48:02.445 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:48:02.445 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:48:02.446 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 0, 2020-03-11T22:48:02.446 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 0, 2020-03-11T22:48:02.446 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 10, 2020-03-11T22:48:02.446 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 24, 2020-03-11T22:48:02.447 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 31, 2020-03-11T22:48:02.447 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:48:02.447 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:02.447 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.447 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.447 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.448 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:48:02.448 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.448 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.448 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.448 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.449 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:48:02.449 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.449 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.449 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.449 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.449 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:48:02.450 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.450 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.450 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.450 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:48:02.450 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.451 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.451 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.451 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.451 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:48:02.451 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.452 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:48:02.453 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.453 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.453 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.453 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.453 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:48:02.453 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.453 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.454 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.454 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.454 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:48:02.454 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.454 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.454 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.454 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.455 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.456 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.456 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/43915", 2020-03-11T22:48:02.456 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/43915", 2020-03-11T22:48:02.456 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43915", 2020-03-11T22:48:02.456 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43915", 2020-03-11T22:48:02.457 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:48:02.457 INFO:tasks.workunit.client.0.smithi142.stdout: "exists" 2020-03-11T22:48:02.457 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.457 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.458 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.458 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:02.458 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "2c7e5aef-f703-4a53-8c31-d0d8c932abf4", 2020-03-11T22:48:02.458 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:48:02.459 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:48:02.459 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:48:02.459 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:48:02.459 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 15, 2020-03-11T22:48:02.459 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 28, 2020-03-11T22:48:02.460 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 36, 2020-03-11T22:48:02.460 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 0, 2020-03-11T22:48:02.460 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 32, 2020-03-11T22:48:02.460 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:48:02.460 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:02.461 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.461 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.461 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.461 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:48:02.461 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.461 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.462 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.462 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.463 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:48:02.463 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.463 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.463 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.463 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.463 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:48:02.463 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.464 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.464 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.464 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:48:02.464 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.464 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.464 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.464 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.465 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.466 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.466 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:48:02.466 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.466 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.466 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.466 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.466 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:48:02.467 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.467 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.467 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.467 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.467 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:48:02.467 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:02.468 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.468 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:02.468 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:48:02.468 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.468 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.468 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.468 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:02.469 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:48:02.469 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 48867 2020-03-11T22:48:02.469 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.469 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.469 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.469 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/48867", 2020-03-11T22:48:02.469 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/48867", 2020-03-11T22:48:02.470 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48867", 2020-03-11T22:48:02.470 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48867", 2020-03-11T22:48:02.470 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:48:02.470 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:48:02.470 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:48:02.470 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.471 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.471 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.471 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_xinfo": [ 2020-03-11T22:48:02.471 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.471 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:02.471 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:47:14.424797+0000", 2020-03-11T22:48:02.471 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:48:02.472 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:48:02.472 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:48:02.472 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:48:02.472 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:46:21.328315+0000", 2020-03-11T22:48:02.473 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:48:02.473 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.473 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.473 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:02.473 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:47:14.934930+0000", 2020-03-11T22:48:02.474 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:48:02.474 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:48:02.474 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:48:02.474 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:48:02.474 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:46:27.718028+0000", 2020-03-11T22:48:02.474 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:48:02.474 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.475 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.475 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:02.475 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:47:15.514777+0000", 2020-03-11T22:48:02.475 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:48:02.475 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:48:02.475 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:48:02.475 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:48:02.476 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:46:33.666441+0000", 2020-03-11T22:48:02.476 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:48:02.476 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.476 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.476 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap": [], 2020-03-11T22:48:02.476 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap_items": [], 2020-03-11T22:48:02.476 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_temp": [], 2020-03-11T22:48:02.477 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_temp": [], 2020-03-11T22:48:02.477 INFO:tasks.workunit.client.0.smithi142.stdout: "blacklist": {}, 2020-03-11T22:48:02.477 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profiles": { 2020-03-11T22:48:02.477 INFO:tasks.workunit.client.0.smithi142.stdout: "default": { 2020-03-11T22:48:02.477 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:48:02.477 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "2", 2020-03-11T22:48:02.477 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:48:02.478 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van" 2020-03-11T22:48:02.478 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.478 INFO:tasks.workunit.client.0.smithi142.stdout: "myprofile": { 2020-03-11T22:48:02.478 INFO:tasks.workunit.client.0.smithi142.stdout: "crush-device-class": "", 2020-03-11T22:48:02.478 INFO:tasks.workunit.client.0.smithi142.stdout: "crush-failure-domain": "osd", 2020-03-11T22:48:02.478 INFO:tasks.workunit.client.0.smithi142.stdout: "crush-root": "default", 2020-03-11T22:48:02.478 INFO:tasks.workunit.client.0.smithi142.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "1", 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van", 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: "w": "8" 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.479 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps_queue": [], 2020-03-11T22:48:02.480 INFO:tasks.workunit.client.0.smithi142.stdout: "new_removed_snaps": [], 2020-03-11T22:48:02.480 INFO:tasks.workunit.client.0.smithi142.stdout: "new_purged_snaps": [], 2020-03-11T22:48:02.480 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_node_flags": {}, 2020-03-11T22:48:02.480 INFO:tasks.workunit.client.0.smithi142.stdout: "device_class_flags": {} 2020-03-11T22:48:02.480 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.480 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_metadata": [ 2020-03-11T22:48:02.480 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.481 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:02.481 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:48:02.481 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6804/48673,v1:127.0.0.1:6805/48673]", 2020-03-11T22:48:02.481 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:48:02.481 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:48:02.481 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:48:02.481 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:48:02.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:48:02.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:48:02.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:48:02.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:48:02.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:48:02.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:48:02.482 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:48:02.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:48:02.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:48:02.483 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:48:02.483 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:48:02.483 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:48:02.483 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:48:02.483 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:48:02.484 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6802/48673,v1:127.0.0.1:6803/48673]", 2020-03-11T22:48:02.485 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:48:02.485 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/48673,v1:127.0.0.1:6809/48673]", 2020-03-11T22:48:02.485 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/48673,v1:127.0.0.1:6807/48673]", 2020-03-11T22:48:02.485 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:48:02.485 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:48:02.485 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:48:02.485 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:48:02.486 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:48:02.486 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:48:02.486 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:48:02.486 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:48:02.486 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:48:02.486 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:48:02.486 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/43915,v1:127.0.0.1:6813/43915]", 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:48:02.487 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:48:02.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:48:02.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:48:02.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:48:02.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:48:02.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:48:02.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:48:02.488 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:48:02.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:48:02.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:48:02.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:48:02.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:48:02.489 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:48:02.489 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:48:02.489 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:48:02.490 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:48:02.491 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:48:02.491 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/43915,v1:127.0.0.1:6811/43915]", 2020-03-11T22:48:02.491 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:48:02.491 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43915,v1:127.0.0.1:6817/43915]", 2020-03-11T22:48:02.491 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43915,v1:127.0.0.1:6815/43915]", 2020-03-11T22:48:02.491 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:48:02.491 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:48:02.492 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:48:02.492 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:48:02.492 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:48:02.492 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:48:02.492 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:48:02.492 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:48:02.492 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:48:02.493 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/48867,v1:127.0.0.1:6813/48867]", 2020-03-11T22:48:02.494 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:48:02.494 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:48:02.494 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:48:02.494 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:48:02.494 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:48:02.494 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:48:02.494 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:48:02.495 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:48:02.495 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:48:02.495 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:48:02.495 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:48:02.495 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:48:02.495 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:48:02.495 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:48:02.496 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:48:02.497 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:48:02.497 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:48:02.497 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:48:02.497 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/48867,v1:127.0.0.1:6811/48867]", 2020-03-11T22:48:02.497 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:48:02.497 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48867,v1:127.0.0.1:6817/48867]", 2020-03-11T22:48:02.497 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48867,v1:127.0.0.1:6815/48867]", 2020-03-11T22:48:02.498 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:48:02.498 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:48:02.498 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:48:02.498 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:48:02.498 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:48:02.498 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:48:02.498 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:48:02.499 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:48:02.499 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:48:02.499 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:48:02.499 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:48:02.499 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:48:02.499 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.499 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.500 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_first_committed": 1, 2020-03-11T22:48:02.500 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_last_committed": 39, 2020-03-11T22:48:02.500 INFO:tasks.workunit.client.0.smithi142.stdout: "crushmap": { 2020-03-11T22:48:02.500 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": [ 2020-03-11T22:48:02.500 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.500 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:02.500 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.0", 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.1", 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.501 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.502 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:02.502 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.2", 2020-03-11T22:48:02.502 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:48:02.502 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.502 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.502 INFO:tasks.workunit.client.0.smithi142.stdout: "types": [ 2020-03-11T22:48:02.502 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.503 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 0, 2020-03-11T22:48:02.503 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd" 2020-03-11T22:48:02.503 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.503 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.503 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:48:02.503 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "host" 2020-03-11T22:48:02.503 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.504 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.504 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 2, 2020-03-11T22:48:02.504 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "chassis" 2020-03-11T22:48:02.504 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.504 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.504 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 3, 2020-03-11T22:48:02.504 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "rack" 2020-03-11T22:48:02.505 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.505 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.505 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 4, 2020-03-11T22:48:02.505 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "row" 2020-03-11T22:48:02.505 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.505 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.505 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 5, 2020-03-11T22:48:02.506 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pdu" 2020-03-11T22:48:02.506 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.506 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.506 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 6, 2020-03-11T22:48:02.506 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pod" 2020-03-11T22:48:02.506 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.506 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.507 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 7, 2020-03-11T22:48:02.507 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "room" 2020-03-11T22:48:02.507 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.507 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.507 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 8, 2020-03-11T22:48:02.507 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "datacenter" 2020-03-11T22:48:02.508 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.508 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.508 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 9, 2020-03-11T22:48:02.508 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "zone" 2020-03-11T22:48:02.508 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.508 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.508 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 10, 2020-03-11T22:48:02.509 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "region" 2020-03-11T22:48:02.509 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.509 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.509 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:48:02.509 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "root" 2020-03-11T22:48:02.509 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.509 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.510 INFO:tasks.workunit.client.0.smithi142.stdout: "buckets": [ 2020-03-11T22:48:02.510 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.510 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -1, 2020-03-11T22:48:02.510 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default", 2020-03-11T22:48:02.510 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:48:02.510 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:48:02.510 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:02.511 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.512 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.512 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.512 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.512 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -2, 2020-03-11T22:48:02.512 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default~hdd", 2020-03-11T22:48:02.512 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:48:02.512 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:48:02.513 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:02.513 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:02.513 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:02.513 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:02.513 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.513 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:48:02.513 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:02.514 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:02.514 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.514 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.514 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.514 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.514 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:48:02.514 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142", 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.515 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:02.516 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:02.516 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:02.516 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.516 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.516 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:02.516 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:02.516 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:48:02.517 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.517 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.517 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:02.517 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:02.517 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:48:02.517 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.517 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142~hdd", 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:02.518 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:02.519 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:02.519 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:02.519 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.519 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:02.519 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:02.519 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:02.519 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.520 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.520 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:02.520 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:02.520 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:48:02.520 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.520 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.520 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:02.521 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:02.521 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:48:02.521 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.521 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.521 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.521 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.521 INFO:tasks.workunit.client.0.smithi142.stdout: "rules": [ 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 0, 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "replicated_rule", 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 0, 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 1, 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 10, 2020-03-11T22:48:02.522 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:48:02.523 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.523 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:48:02.523 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:48:02.523 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:48:02.523 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.523 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.523 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_firstn", 2020-03-11T22:48:02.524 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:48:02.524 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:48:02.524 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.524 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.524 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:48:02.524 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.524 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 1, 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "test", 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 1, 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 3, 2020-03-11T22:48:02.525 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 3, 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 3, 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 5 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.526 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "set_choose_tries", 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 100 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:48:02.527 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_indep", 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:48:02.528 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.529 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:02.529 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.529 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.529 INFO:tasks.workunit.client.0.smithi142.stdout: "tunables": { 2020-03-11T22:48:02.529 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_tries": 0, 2020-03-11T22:48:02.529 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:48:02.529 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_total_tries": 50, 2020-03-11T22:48:02.530 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:48:02.530 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:48:02.530 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_stable": 1, 2020-03-11T22:48:02.530 INFO:tasks.workunit.client.0.smithi142.stdout: "straw_calc_version": 1, 2020-03-11T22:48:02.530 INFO:tasks.workunit.client.0.smithi142.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:48:02.530 INFO:tasks.workunit.client.0.smithi142.stdout: "profile": "jewel", 2020-03-11T22:48:02.530 INFO:tasks.workunit.client.0.smithi142.stdout: "optimal_tunables": 1, 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "legacy_tunables": 0, 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "minimum_required_version": "jewel", 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables": 1, 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables2": 1, 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v2_rules": 1, 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables3": 1, 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v3_rules": 0, 2020-03-11T22:48:02.531 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v4_buckets": 1, 2020-03-11T22:48:02.532 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables5": 1, 2020-03-11T22:48:02.532 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v5_rules": 0 2020-03-11T22:48:02.532 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.532 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_args": {} 2020-03-11T22:48:02.532 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.532 INFO:tasks.workunit.client.0.smithi142.stdout: "fsmap": { 2020-03-11T22:48:02.532 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:48:02.533 INFO:tasks.workunit.client.0.smithi142.stdout: "default_fscid": -1, 2020-03-11T22:48:02.533 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": { 2020-03-11T22:48:02.533 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": {}, 2020-03-11T22:48:02.533 INFO:tasks.workunit.client.0.smithi142.stdout: "ro_compat": {}, 2020-03-11T22:48:02.533 INFO:tasks.workunit.client.0.smithi142.stdout: "incompat": { 2020-03-11T22:48:02.533 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_1": "base v0.20", 2020-03-11T22:48:02.533 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_8": "no anchor table", 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_9": "file layout v2", 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:48:02.534 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_flags": { 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: "enable_multiple": false, 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: "ever_enabled_multiple": false 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: "standbys": [], 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: "filesystems": [] 2020-03-11T22:48:02.535 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.536 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:48:02.536 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:48:02.536 INFO:tasks.workunit.client.0.smithi142.stdout: "auth": { 2020-03-11T22:48:02.536 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:48:02.536 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 5, 2020-03-11T22:48:02.536 INFO:tasks.workunit.client.0.smithi142.stdout: "num_secrets": 3 2020-03-11T22:48:02.536 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.537 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:48:02.537 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_active": 1, 2020-03-11T22:48:02.537 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_unknown": 0, 2020-03-11T22:48:02.537 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osd": 3, 2020-03-11T22:48:02.537 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_sum": { 2020-03-11T22:48:02.537 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:48:02.537 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48430, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 101, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 1, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 303, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 101, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:48:02.538 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:48:02.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 101, 2020-03-11T22:48:02.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:48:02.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:48:02.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:48:02.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:48:02.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:48:02.539 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:48:02.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:48:02.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:48:02.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:48:02.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:48:02.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:48:02.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:48:02.540 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:48:02.541 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:48:02.541 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:48:02.541 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:48:02.541 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:48:02.541 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:48:02.541 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:48:02.541 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:48:02.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:48:02.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:48:02.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:48:02.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:48:02.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:48:02.542 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:48:02.543 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:48:02.543 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:48:02.543 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:48:02.543 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:48:02.543 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.543 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:48:02.543 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:48:02.544 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:48:02.544 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:48:02.544 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 0, 2020-03-11T22:48:02.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 0, 2020-03-11T22:48:02.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:02.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:02.544 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:02.545 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:48:02.545 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:48:02.545 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.545 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 18, 2020-03-11T22:48:02.545 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 18, 2020-03-11T22:48:02.545 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 3, 2020-03-11T22:48:02.545 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 3, 2020-03-11T22:48:02.546 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 0 2020-03-11T22:48:02.546 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.546 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum": { 2020-03-11T22:48:02.546 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:48:02.546 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:48:02.546 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:48:02.547 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:48:02.547 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:48:02.547 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:48:02.547 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:48:02.547 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3179456, 2020-03-11T22:48:02.547 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 33472, 2020-03-11T22:48:02.547 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 11, 2020-03-11T22:48:02.548 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145716, 2020-03-11T22:48:02.548 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393344, 2020-03-11T22:48:02.548 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:48:02.548 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:48:02.548 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318866784256, 2020-03-11T22:48:02.548 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:48:02.548 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 34275328, 2020-03-11T22:48:02.549 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 1517052, 2020-03-11T22:48:02.549 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:02.549 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:02.549 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:02.549 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 11500, 2020-03-11T22:48:02.549 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221213972 2020-03-11T22:48:02.549 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:48:02.550 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.551 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:48:02.551 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:48:02.551 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:48:02.551 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:48:02.551 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:48:02.551 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.551 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:48:02.552 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:48:02.552 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.552 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum_by_class": { 2020-03-11T22:48:02.552 INFO:tasks.workunit.client.0.smithi142.stdout: "hdd": { 2020-03-11T22:48:02.552 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:48:02.552 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:48:02.552 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:48:02.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:48:02.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:48:02.553 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:48:02.553 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:48:02.553 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3179456, 2020-03-11T22:48:02.553 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 33472, 2020-03-11T22:48:02.553 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 11, 2020-03-11T22:48:02.554 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145716, 2020-03-11T22:48:02.554 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311393344, 2020-03-11T22:48:02.554 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:48:02.554 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:48:02.554 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318866784256, 2020-03-11T22:48:02.554 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:48:02.554 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 34275328, 2020-03-11T22:48:02.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 1517052, 2020-03-11T22:48:02.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:02.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:02.555 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:02.555 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 11500, 2020-03-11T22:48:02.555 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221213972 2020-03-11T22:48:02.555 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:48:02.556 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.557 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:48:02.557 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:48:02.557 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:48:02.557 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:48:02.557 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:48:02.557 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.557 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:48:02.558 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:48:02.558 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.558 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.558 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_stats": [ 2020-03-11T22:48:02.558 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.558 INFO:tasks.workunit.client.0.smithi142.stdout: "poolid": 1, 2020-03-11T22:48:02.558 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48430, 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 101, 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 1, 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 303, 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:48:02.559 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 101, 2020-03-11T22:48:02.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:48:02.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:48:02.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 101, 2020-03-11T22:48:02.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:48:02.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:48:02.560 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:48:02.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:48:02.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:48:02.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:48:02.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:48:02.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:48:02.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:48:02.561 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:48:02.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:48:02.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:48:02.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:48:02.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:48:02.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:48:02.562 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:48:02.563 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:48:02.563 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:48:02.563 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:48:02.563 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:48:02.563 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:48:02.563 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:48:02.563 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:48:02.564 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:48:02.564 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:48:02.564 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:48:02.564 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:48:02.564 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:48:02.564 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:48:02.564 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:48:02.565 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.565 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:48:02.565 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:48:02.565 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:48:02.565 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:48:02.565 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 19857408, 2020-03-11T22:48:02.565 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 1241088, 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 18, 2020-03-11T22:48:02.566 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 18, 2020-03-11T22:48:02.567 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 3, 2020-03-11T22:48:02.567 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 3, 2020-03-11T22:48:02.567 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 3 2020-03-11T22:48:02.567 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.567 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.567 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_stats": [ 2020-03-11T22:48:02.567 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.568 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:02.568 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 150323855371 2020-03-11T22:48:02.568 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.568 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.568 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:02.568 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 42949672979 2020-03-11T22:48:02.568 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 154618822666 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_state": [ 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.569 INFO:tasks.workunit.client.0.smithi142.stdout: "state": "active+undersized+degraded", 2020-03-11T22:48:02.570 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 1 2020-03-11T22:48:02.570 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.570 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.570 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_osd": [ 2020-03-11T22:48:02.570 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.570 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:02.570 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2147483647, 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:48:02.571 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:02.572 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.572 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.572 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:02.572 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:48:02.572 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 0, 2020-03-11T22:48:02.572 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:02.572 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.573 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.573 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:02.573 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 1, 2020-03-11T22:48:02.573 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:48:02.573 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:02.573 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.573 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.574 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": [ 2020-03-11T22:48:02.574 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:02.574 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:48:02.574 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": {} 2020-03-11T22:48:02.574 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.574 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:02.574 INFO:tasks.workunit.client.0.smithi142.stdout: "servicemap": { 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:46:14.433718+0000", 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: "services": {} 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: "paxos": { 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 327, 2020-03-11T22:48:02.575 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pn": 0, 2020-03-11T22:48:02.576 INFO:tasks.workunit.client.0.smithi142.stdout: "accepted_pn": 0 2020-03-11T22:48:02.576 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:02.576 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:48:02.578 INFO:tasks.workunit.client.0.smithi142.stderr:report 140301613 2020-03-11T22:48:02.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:48:02.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:48:02.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:48:02.578 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:02.578 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:02.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:02.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:02.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:02.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:02.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:48:02.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:48:02.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:48:02.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:48:02.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:48:02.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:48:02.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:48:03.022 INFO:tasks.workunit.client.0.smithi142.stderr:PG '1.0' not found 2020-03-11T22:48:03.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:48:03.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:48:03.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:48:03.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:48:03.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:48:03.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:48:03.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:48:03.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:48:03.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:03.541 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:03.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:03.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:48:03.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:48:03.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:03.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:48:03.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:48:03.542 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:48:03.543 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:03.543 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:03.543 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:03.543 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:03.543 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:48:03.543 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:48:03.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:48:03.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:03.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:48:03.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:48:03.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:03.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:03.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:03.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:48:03.545 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:48:03.545 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:48:03.545 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:48:03.545 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:48:03.546 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:03.546 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:48:03.546 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:48:03.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:48:03.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:48:03.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:48:03.570 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:03.567+0000 7ff3f0b02ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:03.572 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:03.569+0000 7ff3f0b02ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:03.575 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:03.572+0000 7ff3f0b02ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:03.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:48:03.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:03.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:48:03.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:03.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:03.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:03.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:03.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:03.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:03.898 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:48:04.091 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:04.089+0000 7ff3f0b02ec0 -1 Falling back to public interface 2020-03-11T22:48:04.249 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/48673,v1:127.0.0.1:6803/48673] [v2:127.0.0.1:6804/48673,v1:127.0.0.1:6805/48673] exists,up 36e0b51c-dec2-42db-b07e-d9cf10df314e 2020-03-11T22:48:04.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:04.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:04.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:04.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:48:04.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:48:04.250 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:48:04.250 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:48:04.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:48:04.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:48:04.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:48:04.353 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:48:04.353 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:48:04.353 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:48:04.353 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:48:04.353 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:48:04.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:48:04.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:48:04.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:48:05.011 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:05.008+0000 7ff3f0b02ec0 -1 osd.0 39 log_to_monitors {default=true} 2020-03-11T22:48:05.069 INFO:tasks.workunit.client.0.smithi142.stderr:PG '1.0' not found 2020-03-11T22:48:05.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:48:05.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:48:05.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:48:05.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:48:05.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:48:05.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:48:05.587 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:48:05.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:48:05.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:05.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:05.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:05.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:48:05.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:48:05.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:05.589 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:48:05.590 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:48:05.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:48:05.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:05.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:48:05.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:48:05.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:05.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:05.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:05.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:48:05.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:48:05.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:48:05.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:48:05.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:05.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:48:05.592 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:48:05.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:48:05.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:48:05.593 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:48:05.593 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:48:05.614 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:05.611+0000 7f32a1554ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:05.616 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:05.613+0000 7f32a1554ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:05.618 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:05.615+0000 7f32a1554ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:05.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:48:05.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:05.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:48:05.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:05.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:05.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:05.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:05.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:05.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:05.936 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:48:06.133 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:06.130+0000 7f32a1554ec0 -1 Falling back to public interface 2020-03-11T22:48:06.280 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/48867,v1:127.0.0.1:6811/48867] [v2:127.0.0.1:6812/48867,v1:127.0.0.1:6813/48867] exists,up 2c7e5aef-f703-4a53-8c31-d0d8c932abf4 2020-03-11T22:48:06.280 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:06.280 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:06.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:06.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:48:06.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:48:06.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:48:06.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:48:06.282 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:48:06.282 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:48:06.282 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:48:06.896 INFO:tasks.workunit.client.0.smithi142.stderr:PG '1.0' not found 2020-03-11T22:48:07.030 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:07.028+0000 7f32a1554ec0 -1 osd.2 39 log_to_monitors {default=true} 2020-03-11T22:48:07.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:48:07.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:48:07.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:48:07.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:48:07.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:48:07.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:48:07.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:48:07.415 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:48:07.415 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:48:07.415 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:48:07.416 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:48:07.416 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:48:07.416 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:48:07.477 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:48:07.477 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:48:07.477 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:48:07.477 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:48:07.477 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:48:07.825 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:48:07.826 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:48:07.826 INFO:tasks.workunit.client.0.smithi142.stderr:2' 2020-03-11T22:48:07.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:48:07.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:07.826 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:48:07.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=176093659139 2020-03-11T22:48:07.954 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139' 2020-03-11T22:48:07.955 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:07.955 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:48:08.046 INFO:tasks.workunit.client.0.smithi142.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:48:08.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:48:08.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139 1-' 2020-03-11T22:48:08.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:08.054 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:48:08.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=188978561027 2020-03-11T22:48:08.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-176093659139 1- 2-188978561027' 2020-03-11T22:48:08.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:08.184 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-176093659139 2020-03-11T22:48:08.184 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:08.184 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:48:08.185 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-176093659139 2020-03-11T22:48:08.185 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:08.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=176093659139 2020-03-11T22:48:08.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 176093659139' 2020-03-11T22:48:08.186 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:08.186 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 176093659139 2020-03-11T22:48:08.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 176093659139 2020-03-11T22:48:08.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:09.529 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:48:09.530 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:09.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 176093659139 -lt 176093659139 2020-03-11T22:48:09.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:09.898 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:48:09.898 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:09.899 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:48:09.899 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:48:09.899 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:09.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:48:09.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:48:09.901 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:48:09.901 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 2020-03-11T22:48:10.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672979 -lt 2020-03-11T22:48:10.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672979: unary operator expected 2020-03-11T22:48:10.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:10.270 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-188978561027 2020-03-11T22:48:10.270 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:10.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:48:10.271 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-188978561027 2020-03-11T22:48:10.271 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:10.272 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=188978561027 2020-03-11T22:48:10.272 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 188978561027' 2020-03-11T22:48:10.272 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:48:10.273 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 188978561027 2020-03-11T22:48:10.613 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 188978561027 -lt 188978561027 2020-03-11T22:48:10.614 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:48:10.614 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:10.614 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:11.057 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:48:11.058 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:11.059 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:11.059 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:11.059 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:11.059 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:11.059 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:11.059 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:11.390 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:11.390 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:11.391 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:11.391 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:11.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:11.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:48:11.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:48:11.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:48:11.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:11.837 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:11.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:11.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:11.937 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:11.937 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:11.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:11.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:11.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:11.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:12.267 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:12.268 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:12.268 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:12.268 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:12.719 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:12.719 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:12.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:12.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:12.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:12.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:12.720 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:12.720 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:12.720 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:13.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:13.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:13.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:48:13.168 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:13.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:48:13.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:13.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:13.370 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:13.370 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:13.370 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:13.370 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:13.370 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:13.370 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:13.700 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:13.700 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:13.701 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:13.701 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:14.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:14.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:14.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:14.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:14.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:14.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:14.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:14.143 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:14.143 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:14.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:14.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:14.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:48:14.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:14.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:48:14.984 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:14.985 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:14.986 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:14.986 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:14.986 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:14.986 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:14.986 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:14.987 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:15.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:15.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:15.319 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:15.319 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:15.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:15.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:15.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:15.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:15.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:15.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:15.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:15.762 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:15.762 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:16.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:16.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:16.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:48:16.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:16.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:48:17.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:17.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:17.013 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:17.013 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:17.014 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:17.014 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:17.014 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:17.014 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:17.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:17.355 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:17.356 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:17.356 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:17.799 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:17.799 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:17.799 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:17.799 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:17.800 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:17.800 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:17.800 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:17.800 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:17.800 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:18.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:18.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:18.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:48:18.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:18.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:48:19.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:19.850 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:19.851 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:19.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:19.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:19.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:19.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:19.853 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:20.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:20.195 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:20.195 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:20.195 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:20.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:20.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:20.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:20.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:20.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:20.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:20.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:20.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:20.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:21.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:21.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:21.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:48:21.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:21.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:48:24.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:24.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:24.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:24.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:24.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:24.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:24.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:24.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:24.662 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:24.662 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:24.663 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:24.663 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:25.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:25.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:25.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:25.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:25.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:25.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:25.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:25.105 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:25.105 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:25.546 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:25.546 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:25.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:48:25.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:25.547 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:48:31.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:31.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:31.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:31.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:31.949 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:31.949 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:31.949 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:31.949 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:32.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:32.293 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:32.293 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:32.293 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:32.740 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:32.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:32.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:32.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:32.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:32.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:32.742 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:32.742 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:32.742 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:33.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:33.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:33.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:48:33.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:33.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:48:40.487 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:40.487 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:40.488 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:40.488 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:40.488 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:40.488 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:40.488 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:40.489 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:40.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:40.829 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:40.829 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:40.829 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:41.268 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:48:41.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:41.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:41.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:41.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:41.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:41.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:41.269 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:41.270 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:41.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:48:41.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:48:41.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:48:41.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:48:42.149 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:48:42.149 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_fingerprint": "c22feccd-ec9a-415b-bb61-741d5acf9558", 2020-03-11T22:48:42.149 INFO:tasks.workunit.client.0.smithi142.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:48:42.149 INFO:tasks.workunit.client.0.smithi142.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:48:42.150 INFO:tasks.workunit.client.0.smithi142.stdout: "timestamp": "2020-03-11T22:48:42.145748+0000", 2020-03-11T22:48:42.150 INFO:tasks.workunit.client.0.smithi142.stdout: "tag": "", 2020-03-11T22:48:42.150 INFO:tasks.workunit.client.0.smithi142.stdout: "health": { 2020-03-11T22:48:42.150 INFO:tasks.workunit.client.0.smithi142.stdout: "status": "HEALTH_WARN", 2020-03-11T22:48:42.150 INFO:tasks.workunit.client.0.smithi142.stdout: "checks": { 2020-03-11T22:48:42.150 INFO:tasks.workunit.client.0.smithi142.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:48:42.150 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 17 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [], 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.151 INFO:tasks.workunit.client.0.smithi142.stdout: "OSD_DOWN": { 2020-03-11T22:48:42.152 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:42.152 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:42.152 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 osds down", 2020-03-11T22:48:42.152 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:48:42.152 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.152 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:48:42.154 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.155 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "osd.1 (root=default,host=smithi142) is down" 2020-03-11T22:48:42.155 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.155 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.155 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:42.155 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.155 INFO:tasks.workunit.client.0.smithi142.stdout: "PG_DEGRADED": { 2020-03-11T22:48:42.156 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:42.156 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:42.156 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:48:42.156 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:48:42.157 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.157 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:48:42.157 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.157 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:48:42.157 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.157 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.157 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: "summary": { 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: "count": 1 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.158 INFO:tasks.workunit.client.0.smithi142.stdout: "detail": [ 2020-03-11T22:48:42.159 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.159 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:48:42.159 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.159 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.159 INFO:tasks.workunit.client.0.smithi142.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:48:42.159 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.159 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.160 INFO:tasks.workunit.client.0.smithi142.stdout: "muted": false 2020-03-11T22:48:42.160 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.160 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.160 INFO:tasks.workunit.client.0.smithi142.stdout: "mutes": [] 2020-03-11T22:48:42.160 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.160 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_first_committed": 1, 2020-03-11T22:48:42.160 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap_last_committed": 1, 2020-03-11T22:48:42.161 INFO:tasks.workunit.client.0.smithi142.stdout: "monmap": { 2020-03-11T22:48:42.161 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:48:42.161 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:48:42.161 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:46:13.742457Z", 2020-03-11T22:48:42.161 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:46:13.742457Z", 2020-03-11T22:48:42.161 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release": 15, 2020-03-11T22:48:42.161 INFO:tasks.workunit.client.0.smithi142.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "features": { 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "persistent": [ 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "kraken", 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "luminous", 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "mimic", 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap-prune", 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "nautilus", 2020-03-11T22:48:42.162 INFO:tasks.workunit.client.0.smithi142.stdout: "octopus" 2020-03-11T22:48:42.163 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.163 INFO:tasks.workunit.client.0.smithi142.stdout: "optional": [] 2020-03-11T22:48:42.163 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.163 INFO:tasks.workunit.client.0.smithi142.stdout: "mons": [ 2020-03-11T22:48:42.163 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.163 INFO:tasks.workunit.client.0.smithi142.stdout: "rank": 0, 2020-03-11T22:48:42.163 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "a", 2020-03-11T22:48:42.165 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:42.165 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.166 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.166 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.166 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:48:42.167 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 0 2020-03-11T22:48:42.167 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.167 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.167 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.167 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:48:42.167 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:48:42.168 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 0, 2020-03-11T22:48:42.168 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 0 2020-03-11T22:48:42.168 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.168 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.168 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.168 INFO:tasks.workunit.client.0.smithi142.stdout: "quorum": [ 2020-03-11T22:48:42.168 INFO:tasks.workunit.client.0.smithi142.stdout: 0 2020-03-11T22:48:42.169 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.169 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap": { 2020-03-11T22:48:42.169 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 45, 2020-03-11T22:48:42.169 INFO:tasks.workunit.client.0.smithi142.stdout: "fsid": "e4d81029-c251-4939-a03a-37f830550d7a", 2020-03-11T22:48:42.169 INFO:tasks.workunit.client.0.smithi142.stdout: "created": "2020-03-11T22:46:14.433398+0000", 2020-03-11T22:48:42.169 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:48:07.387194+0000", 2020-03-11T22:48:42.169 INFO:tasks.workunit.client.0.smithi142.stdout: "last_up_change": "2020-03-11T22:48:07.245183+0000", 2020-03-11T22:48:42.170 INFO:tasks.workunit.client.0.smithi142.stdout: "last_in_change": "2020-03-11T22:47:21.998697+0000", 2020-03-11T22:48:42.170 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:48:42.170 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_num": 5800384, 2020-03-11T22:48:42.170 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_set": [ 2020-03-11T22:48:42.170 INFO:tasks.workunit.client.0.smithi142.stdout: "nodown", 2020-03-11T22:48:42.170 INFO:tasks.workunit.client.0.smithi142.stdout: "noin", 2020-03-11T22:48:42.170 INFO:tasks.workunit.client.0.smithi142.stdout: "noout", 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: "pglog_hardlimit", 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snapdirs", 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: "recovery_deletes", 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: "sortbitwise" 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_version": 8, 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:48:42.171 INFO:tasks.workunit.client.0.smithi142.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:48:42.172 INFO:tasks.workunit.client.0.smithi142.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:48:42.172 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_snapshot": "", 2020-03-11T22:48:42.172 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_max": 1, 2020-03-11T22:48:42.172 INFO:tasks.workunit.client.0.smithi142.stdout: "max_osd": 3, 2020-03-11T22:48:42.172 INFO:tasks.workunit.client.0.smithi142.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:48:42.172 INFO:tasks.workunit.client.0.smithi142.stdout: "min_compat_client": "jewel", 2020-03-11T22:48:42.172 INFO:tasks.workunit.client.0.smithi142.stdout: "require_osd_release": "octopus", 2020-03-11T22:48:42.173 INFO:tasks.workunit.client.0.smithi142.stdout: "pools": [ 2020-03-11T22:48:42.173 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.173 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:48:42.173 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_name": "test", 2020-03-11T22:48:42.173 INFO:tasks.workunit.client.0.smithi142.stdout: "create_time": "2020-03-11T22:46:36.237276+0000", 2020-03-11T22:48:42.173 INFO:tasks.workunit.client.0.smithi142.stdout: "flags": 16389, 2020-03-11T22:48:42.173 INFO:tasks.workunit.client.0.smithi142.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:48:42.174 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 3, 2020-03-11T22:48:42.174 INFO:tasks.workunit.client.0.smithi142.stdout: "size": 3, 2020-03-11T22:48:42.174 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 2, 2020-03-11T22:48:42.174 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_rule": 1, 2020-03-11T22:48:42.174 INFO:tasks.workunit.client.0.smithi142.stdout: "object_hash": 2, 2020-03-11T22:48:42.174 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:48:42.175 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num": 1, 2020-03-11T22:48:42.175 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num": 1, 2020-03-11T22:48:42.175 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_placement_num_target": 1, 2020-03-11T22:48:42.175 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_target": 1, 2020-03-11T22:48:42.175 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_num_pending": 1, 2020-03-11T22:48:42.175 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pg_merge_meta": { 2020-03-11T22:48:42.175 INFO:tasks.workunit.client.0.smithi142.stdout: "source_pgid": "0.0", 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: "ready_epoch": 0, 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_started": 0, 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: "last_epoch_clean": 0, 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: "source_version": "0'0", 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: "target_version": "0'0" 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: "last_change": "29", 2020-03-11T22:48:42.176 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend": "0", 2020-03-11T22:48:42.177 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:48:42.177 INFO:tasks.workunit.client.0.smithi142.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:48:42.177 INFO:tasks.workunit.client.0.smithi142.stdout: "auid": 0, 2020-03-11T22:48:42.177 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_mode": "pool", 2020-03-11T22:48:42.177 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_seq": 1, 2020-03-11T22:48:42.177 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_epoch": 29, 2020-03-11T22:48:42.177 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_snaps": [ 2020-03-11T22:48:42.178 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.178 INFO:tasks.workunit.client.0.smithi142.stdout: "snapid": 1, 2020-03-11T22:48:42.178 INFO:tasks.workunit.client.0.smithi142.stdout: "stamp": "2020-03-11T22:47:03.541351+0000", 2020-03-11T22:48:42.178 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "snap1" 2020-03-11T22:48:42.178 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.178 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.179 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps": "[]", 2020-03-11T22:48:42.179 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_bytes": 0, 2020-03-11T22:48:42.179 INFO:tasks.workunit.client.0.smithi142.stdout: "quota_max_objects": 0, 2020-03-11T22:48:42.180 INFO:tasks.workunit.client.0.smithi142.stdout: "tiers": [], 2020-03-11T22:48:42.180 INFO:tasks.workunit.client.0.smithi142.stdout: "tier_of": -1, 2020-03-11T22:48:42.180 INFO:tasks.workunit.client.0.smithi142.stdout: "read_tier": -1, 2020-03-11T22:48:42.180 INFO:tasks.workunit.client.0.smithi142.stdout: "write_tier": -1, 2020-03-11T22:48:42.180 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_mode": "none", 2020-03-11T22:48:42.180 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_bytes": 0, 2020-03-11T22:48:42.180 INFO:tasks.workunit.client.0.smithi142.stdout: "target_max_objects": 0, 2020-03-11T22:48:42.181 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:48:42.181 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:48:42.181 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:48:42.181 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_flush_age": 0, 2020-03-11T22:48:42.181 INFO:tasks.workunit.client.0.smithi142.stdout: "cache_min_evict_age": 0, 2020-03-11T22:48:42.181 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:48:42.181 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_params": { 2020-03-11T22:48:42.182 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "none" 2020-03-11T22:48:42.182 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.182 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_period": 0, 2020-03-11T22:48:42.182 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_count": 0, 2020-03-11T22:48:42.182 INFO:tasks.workunit.client.0.smithi142.stdout: "use_gmt_hitset": true, 2020-03-11T22:48:42.182 INFO:tasks.workunit.client.0.smithi142.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:48:42.182 INFO:tasks.workunit.client.0.smithi142.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:48:42.183 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:48:42.183 INFO:tasks.workunit.client.0.smithi142.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:48:42.183 INFO:tasks.workunit.client.0.smithi142.stdout: "grade_table": [], 2020-03-11T22:48:42.183 INFO:tasks.workunit.client.0.smithi142.stdout: "stripe_width": 8192, 2020-03-11T22:48:42.183 INFO:tasks.workunit.client.0.smithi142.stdout: "expected_num_objects": 0, 2020-03-11T22:48:42.183 INFO:tasks.workunit.client.0.smithi142.stdout: "fast_read": false, 2020-03-11T22:48:42.184 INFO:tasks.workunit.client.0.smithi142.stdout: "options": {}, 2020-03-11T22:48:42.184 INFO:tasks.workunit.client.0.smithi142.stdout: "application_metadata": {} 2020-03-11T22:48:42.184 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.184 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.184 INFO:tasks.workunit.client.0.smithi142.stdout: "osds": [ 2020-03-11T22:48:42.184 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.184 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:42.185 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "36e0b51c-dec2-42db-b07e-d9cf10df314e", 2020-03-11T22:48:42.185 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:48:42.185 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:48:42.185 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:48:42.185 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:48:42.185 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 35, 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 39, 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 41, 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 44, 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 40, 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.186 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.187 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.187 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:48:42.187 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.187 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.187 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.187 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.187 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:48:42.188 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.188 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.188 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.188 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.188 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:48:42.188 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.188 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.189 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.189 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:48:42.189 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.189 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.189 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.189 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.190 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:48:42.190 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.190 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.190 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.190 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.190 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:48:42.190 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.191 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.191 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.191 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:48:42.191 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.191 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.191 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.191 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.192 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.193 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.193 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:48:42.193 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.193 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.193 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.193 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.193 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50308 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6803/50308", 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6805/50308", 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50308", 2020-03-11T22:48:42.194 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50308", 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.195 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "11c7e564-1ae2-4288-8725-ccd9a0814a84", 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 0, 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 0, 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 0, 2020-03-11T22:48:42.196 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 10, 2020-03-11T22:48:42.197 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 24, 2020-03-11T22:48:42.197 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 31, 2020-03-11T22:48:42.197 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:48:42.197 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:42.197 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.197 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.198 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.198 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:48:42.198 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.198 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.198 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.198 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.198 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.199 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.200 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:48:42.200 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.200 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.200 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.200 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.200 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:48:42.200 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.201 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:48:42.202 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.202 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.202 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.202 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.202 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:48:42.202 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.202 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.203 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.203 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.203 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:48:42.203 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.203 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.203 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 43915 2020-03-11T22:48:42.204 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.205 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.205 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.205 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/43915", 2020-03-11T22:48:42.205 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/43915", 2020-03-11T22:48:42.205 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43915", 2020-03-11T22:48:42.205 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43915", 2020-03-11T22:48:42.205 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: "exists" 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: "uuid": "2c7e5aef-f703-4a53-8c31-d0d8c932abf4", 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 1, 2020-03-11T22:48:42.206 INFO:tasks.workunit.client.0.smithi142.stdout: "in": 1, 2020-03-11T22:48:42.207 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 1, 2020-03-11T22:48:42.207 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_affinity": 1, 2020-03-11T22:48:42.207 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_begin": 36, 2020-03-11T22:48:42.207 INFO:tasks.workunit.client.0.smithi142.stdout: "last_clean_end": 39, 2020-03-11T22:48:42.207 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 44, 2020-03-11T22:48:42.207 INFO:tasks.workunit.client.0.smithi142.stdout: "up_thru": 0, 2020-03-11T22:48:42.207 INFO:tasks.workunit.client.0.smithi142.stdout: "down_at": 43, 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: "lost_at": 0, 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addrs": { 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.208 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.209 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.209 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.209 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:48:42.209 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.209 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.209 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.209 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addrs": { 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.210 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.211 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.211 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:48:42.211 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.211 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.211 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.211 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.211 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addrs": { 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.212 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addrs": { 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: "addrvec": [ 2020-03-11T22:48:42.213 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.214 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v2", 2020-03-11T22:48:42.214 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:48:42.214 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.214 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.214 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.214 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "v1", 2020-03-11T22:48:42.214 INFO:tasks.workunit.client.0.smithi142.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: "nonce": 50540 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: "public_addr": "127.0.0.1:6811/50540", 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: "cluster_addr": "127.0.0.1:6813/50540", 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_back_addr": "127.0.0.1:6817/50540", 2020-03-11T22:48:42.215 INFO:tasks.workunit.client.0.smithi142.stdout: "heartbeat_front_addr": "127.0.0.1:6815/50540", 2020-03-11T22:48:42.216 INFO:tasks.workunit.client.0.smithi142.stdout: "state": [ 2020-03-11T22:48:42.216 INFO:tasks.workunit.client.0.smithi142.stdout: "exists", 2020-03-11T22:48:42.216 INFO:tasks.workunit.client.0.smithi142.stdout: "up" 2020-03-11T22:48:42.216 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.216 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.216 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.216 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_xinfo": [ 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:48:05.116644+0000", 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:48:42.217 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:46:21.328315+0000", 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:47:14.934930+0000", 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:48:42.218 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:48:42.219 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:48:42.219 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:46:27.718028+0000", 2020-03-11T22:48:42.219 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:48:42.219 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.219 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.219 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:42.219 INFO:tasks.workunit.client.0.smithi142.stdout: "down_stamp": "2020-03-11T22:48:07.095032+0000", 2020-03-11T22:48:42.220 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_probability": 0, 2020-03-11T22:48:42.220 INFO:tasks.workunit.client.0.smithi142.stdout: "laggy_interval": 0, 2020-03-11T22:48:42.221 INFO:tasks.workunit.client.0.smithi142.stdout: "features": 4540138292836696063, 2020-03-11T22:48:42.221 INFO:tasks.workunit.client.0.smithi142.stdout: "old_weight": 0, 2020-03-11T22:48:42.221 INFO:tasks.workunit.client.0.smithi142.stdout: "last_purged_snaps_scrub": "2020-03-11T22:46:33.666441+0000", 2020-03-11T22:48:42.221 INFO:tasks.workunit.client.0.smithi142.stdout: "dead_epoch": 0 2020-03-11T22:48:42.221 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.221 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap": [], 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_upmap_items": [], 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "pg_temp": [], 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "primary_temp": [], 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "blacklist": {}, 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "erasure_code_profiles": { 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "default": { 2020-03-11T22:48:42.222 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:48:42.223 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "2", 2020-03-11T22:48:42.223 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:48:42.223 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van" 2020-03-11T22:48:42.223 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.223 INFO:tasks.workunit.client.0.smithi142.stdout: "myprofile": { 2020-03-11T22:48:42.223 INFO:tasks.workunit.client.0.smithi142.stdout: "crush-device-class": "", 2020-03-11T22:48:42.223 INFO:tasks.workunit.client.0.smithi142.stdout: "crush-failure-domain": "osd", 2020-03-11T22:48:42.224 INFO:tasks.workunit.client.0.smithi142.stdout: "crush-root": "default", 2020-03-11T22:48:42.224 INFO:tasks.workunit.client.0.smithi142.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:48:42.224 INFO:tasks.workunit.client.0.smithi142.stdout: "k": "2", 2020-03-11T22:48:42.224 INFO:tasks.workunit.client.0.smithi142.stdout: "m": "1", 2020-03-11T22:48:42.224 INFO:tasks.workunit.client.0.smithi142.stdout: "plugin": "jerasure", 2020-03-11T22:48:42.224 INFO:tasks.workunit.client.0.smithi142.stdout: "technique": "reed_sol_van", 2020-03-11T22:48:42.224 INFO:tasks.workunit.client.0.smithi142.stdout: "w": "8" 2020-03-11T22:48:42.225 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.225 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.225 INFO:tasks.workunit.client.0.smithi142.stdout: "removed_snaps_queue": [], 2020-03-11T22:48:42.225 INFO:tasks.workunit.client.0.smithi142.stdout: "new_removed_snaps": [], 2020-03-11T22:48:42.225 INFO:tasks.workunit.client.0.smithi142.stdout: "new_purged_snaps": [], 2020-03-11T22:48:42.225 INFO:tasks.workunit.client.0.smithi142.stdout: "crush_node_flags": {}, 2020-03-11T22:48:42.225 INFO:tasks.workunit.client.0.smithi142.stdout: "device_class_flags": {} 2020-03-11T22:48:42.226 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.226 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_metadata": [ 2020-03-11T22:48:42.226 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.226 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:42.226 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:48:42.226 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6804/50308,v1:127.0.0.1:6805/50308]", 2020-03-11T22:48:42.226 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:48:42.227 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:48:42.228 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:48:42.228 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:48:42.228 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:48:42.228 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:48:42.228 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:48:42.228 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:48:42.228 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:48:42.229 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:48:42.230 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:48:42.230 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6802/50308,v1:127.0.0.1:6803/50308]", 2020-03-11T22:48:42.230 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:48:42.230 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50308,v1:127.0.0.1:6809/50308]", 2020-03-11T22:48:42.230 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50308,v1:127.0.0.1:6807/50308]", 2020-03-11T22:48:42.230 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:48:42.231 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:48:42.232 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:48:42.232 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:48:42.232 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:48:42.232 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.232 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.232 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:42.232 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:48:42.233 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/43915,v1:127.0.0.1:6813/43915]", 2020-03-11T22:48:42.233 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:48:42.233 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:48:42.233 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:48:42.233 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:48:42.233 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:48:42.234 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:48:42.234 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:48:42.234 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:48:42.234 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:48:42.234 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:48:42.234 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:48:42.234 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:48:42.235 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:48:42.236 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:48:42.236 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:48:42.236 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:48:42.236 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:48:42.236 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:48:42.236 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/43915,v1:127.0.0.1:6811/43915]", 2020-03-11T22:48:42.236 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43915,v1:127.0.0.1:6817/43915]", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43915,v1:127.0.0.1:6815/43915]", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:48:42.237 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:48:42.238 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:48:42.238 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:48:42.238 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:48:42.238 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:48:42.238 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:48:42.238 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:48:42.238 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: "arch": "x86_64", 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: "back_addr": "[v2:127.0.0.1:6812/50540,v1:127.0.0.1:6813/50540]", 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: "back_iface": "", 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs": "1", 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:48:42.239 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:48:42.240 INFO:tasks.workunit.client.0.smithi142.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:48:42.240 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:48:42.240 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:48:42.240 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:48:42.240 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:48:42.240 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:48:42.241 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:48:42.241 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:48:42.241 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:48:42.241 INFO:tasks.workunit.client.0.smithi142.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:48:42.241 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_release": "octopus", 2020-03-11T22:48:42.241 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:48:42.241 INFO:tasks.workunit.client.0.smithi142.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:48:42.242 INFO:tasks.workunit.client.0.smithi142.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:48:42.242 INFO:tasks.workunit.client.0.smithi142.stdout: "default_device_class": "hdd", 2020-03-11T22:48:42.242 INFO:tasks.workunit.client.0.smithi142.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFVQ", 2020-03-11T22:48:42.242 INFO:tasks.workunit.client.0.smithi142.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:48:42.242 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": "sda", 2020-03-11T22:48:42.242 INFO:tasks.workunit.client.0.smithi142.stdout: "distro": "centos", 2020-03-11T22:48:42.242 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "distro_version": "8", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "front_addr": "[v2:127.0.0.1:6810/50540,v1:127.0.0.1:6811/50540]", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "front_iface": "", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/50540,v1:127.0.0.1:6817/50540]", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/50540,v1:127.0.0.1:6815/50540]", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "hostname": "smithi142", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "journal_rotational": "1", 2020-03-11T22:48:42.243 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:48:42.244 INFO:tasks.workunit.client.0.smithi142.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:48:42.244 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_swap_kb": "0", 2020-03-11T22:48:42.244 INFO:tasks.workunit.client.0.smithi142.stdout: "mem_total_kb": "32655820", 2020-03-11T22:48:42.244 INFO:tasks.workunit.client.0.smithi142.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:48:42.244 INFO:tasks.workunit.client.0.smithi142.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:48:42.244 INFO:tasks.workunit.client.0.smithi142.stdout: "os": "Linux", 2020-03-11T22:48:42.244 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:48:42.245 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:48:42.245 INFO:tasks.workunit.client.0.smithi142.stdout: "rotational": "1" 2020-03-11T22:48:42.245 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.245 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.245 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_first_committed": 1, 2020-03-11T22:48:42.245 INFO:tasks.workunit.client.0.smithi142.stdout: "osdmap_last_committed": 45, 2020-03-11T22:48:42.245 INFO:tasks.workunit.client.0.smithi142.stdout: "crushmap": { 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: "devices": [ 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.0", 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.246 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:42.247 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.1", 2020-03-11T22:48:42.247 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:48:42.247 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.247 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.247 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:42.247 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd.2", 2020-03-11T22:48:42.247 INFO:tasks.workunit.client.0.smithi142.stdout: "class": "hdd" 2020-03-11T22:48:42.248 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.248 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.248 INFO:tasks.workunit.client.0.smithi142.stdout: "types": [ 2020-03-11T22:48:42.248 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.248 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 0, 2020-03-11T22:48:42.248 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "osd" 2020-03-11T22:48:42.249 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.249 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.249 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:48:42.249 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "host" 2020-03-11T22:48:42.249 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.249 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.250 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 2, 2020-03-11T22:48:42.250 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "chassis" 2020-03-11T22:48:42.250 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.250 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.250 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 3, 2020-03-11T22:48:42.250 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "rack" 2020-03-11T22:48:42.250 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 4, 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "row" 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 5, 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pdu" 2020-03-11T22:48:42.251 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.252 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.252 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 6, 2020-03-11T22:48:42.252 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "pod" 2020-03-11T22:48:42.252 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.252 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.252 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 7, 2020-03-11T22:48:42.252 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "room" 2020-03-11T22:48:42.253 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.253 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.253 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 8, 2020-03-11T22:48:42.253 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "datacenter" 2020-03-11T22:48:42.253 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.253 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.254 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 9, 2020-03-11T22:48:42.254 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "zone" 2020-03-11T22:48:42.254 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.254 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.254 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 10, 2020-03-11T22:48:42.255 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "region" 2020-03-11T22:48:42.255 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.255 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.255 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:48:42.255 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "root" 2020-03-11T22:48:42.255 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: "buckets": [ 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -1, 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default", 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:48:42.256 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:42.257 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:42.257 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:42.257 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:42.257 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.257 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:48:42.257 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:42.257 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -2, 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "default~hdd", 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 11, 2020-03-11T22:48:42.258 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "root", 2020-03-11T22:48:42.259 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:42.259 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:42.259 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:42.259 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:42.259 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.259 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:48:42.259 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -3, 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142", 2020-03-11T22:48:42.260 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:48:42.261 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:48:42.261 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:42.261 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:42.261 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:42.261 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:42.261 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.261 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:42.262 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:42.262 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:42.262 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.262 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.262 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:42.262 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:42.262 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.263 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.264 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.264 INFO:tasks.workunit.client.0.smithi142.stdout: "id": -4, 2020-03-11T22:48:42.264 INFO:tasks.workunit.client.0.smithi142.stdout: "name": "smithi142~hdd", 2020-03-11T22:48:42.264 INFO:tasks.workunit.client.0.smithi142.stdout: "type_id": 1, 2020-03-11T22:48:42.264 INFO:tasks.workunit.client.0.smithi142.stdout: "type_name": "host", 2020-03-11T22:48:42.264 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 19206, 2020-03-11T22:48:42.264 INFO:tasks.workunit.client.0.smithi142.stdout: "alg": "straw2", 2020-03-11T22:48:42.265 INFO:tasks.workunit.client.0.smithi142.stdout: "hash": "rjenkins1", 2020-03-11T22:48:42.265 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:48:42.265 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.265 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 0, 2020-03-11T22:48:42.265 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:42.265 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 0 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 1, 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 1 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.266 INFO:tasks.workunit.client.0.smithi142.stdout: "id": 2, 2020-03-11T22:48:42.267 INFO:tasks.workunit.client.0.smithi142.stdout: "weight": 6402, 2020-03-11T22:48:42.267 INFO:tasks.workunit.client.0.smithi142.stdout: "pos": 2 2020-03-11T22:48:42.267 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.267 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.267 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.267 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.268 INFO:tasks.workunit.client.0.smithi142.stdout: "rules": [ 2020-03-11T22:48:42.268 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.269 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 0, 2020-03-11T22:48:42.269 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "replicated_rule", 2020-03-11T22:48:42.269 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 0, 2020-03-11T22:48:42.269 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 1, 2020-03-11T22:48:42.269 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 1, 2020-03-11T22:48:42.269 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 10, 2020-03-11T22:48:42.269 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_firstn", 2020-03-11T22:48:42.270 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:48:42.271 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:48:42.271 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.271 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.271 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:48:42.271 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.271 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.271 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.272 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.272 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_id": 1, 2020-03-11T22:48:42.272 INFO:tasks.workunit.client.0.smithi142.stdout: "rule_name": "test", 2020-03-11T22:48:42.272 INFO:tasks.workunit.client.0.smithi142.stdout: "ruleset": 1, 2020-03-11T22:48:42.272 INFO:tasks.workunit.client.0.smithi142.stdout: "type": 3, 2020-03-11T22:48:42.272 INFO:tasks.workunit.client.0.smithi142.stdout: "min_size": 3, 2020-03-11T22:48:42.272 INFO:tasks.workunit.client.0.smithi142.stdout: "max_size": 3, 2020-03-11T22:48:42.273 INFO:tasks.workunit.client.0.smithi142.stdout: "steps": [ 2020-03-11T22:48:42.273 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.274 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:48:42.274 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 5 2020-03-11T22:48:42.274 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.274 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.274 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "set_choose_tries", 2020-03-11T22:48:42.274 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 100 2020-03-11T22:48:42.274 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.275 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.275 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "take", 2020-03-11T22:48:42.275 INFO:tasks.workunit.client.0.smithi142.stdout: "item": -1, 2020-03-11T22:48:42.275 INFO:tasks.workunit.client.0.smithi142.stdout: "item_name": "default" 2020-03-11T22:48:42.275 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.275 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.275 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "choose_indep", 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 0, 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: "type": "osd" 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: "op": "emit" 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:48:42.276 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.277 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.277 INFO:tasks.workunit.client.0.smithi142.stdout: "tunables": { 2020-03-11T22:48:42.277 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_tries": 0, 2020-03-11T22:48:42.277 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:48:42.277 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_total_tries": 50, 2020-03-11T22:48:42.277 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:48:42.277 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:48:42.278 INFO:tasks.workunit.client.0.smithi142.stdout: "chooseleaf_stable": 1, 2020-03-11T22:48:42.278 INFO:tasks.workunit.client.0.smithi142.stdout: "straw_calc_version": 1, 2020-03-11T22:48:42.278 INFO:tasks.workunit.client.0.smithi142.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:48:42.278 INFO:tasks.workunit.client.0.smithi142.stdout: "profile": "jewel", 2020-03-11T22:48:42.278 INFO:tasks.workunit.client.0.smithi142.stdout: "optimal_tunables": 1, 2020-03-11T22:48:42.278 INFO:tasks.workunit.client.0.smithi142.stdout: "legacy_tunables": 0, 2020-03-11T22:48:42.278 INFO:tasks.workunit.client.0.smithi142.stdout: "minimum_required_version": "jewel", 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables": 1, 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables2": 1, 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v2_rules": 1, 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables3": 1, 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v3_rules": 0, 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v4_buckets": 1, 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "require_feature_tunables5": 1, 2020-03-11T22:48:42.279 INFO:tasks.workunit.client.0.smithi142.stdout: "has_v5_rules": 0 2020-03-11T22:48:42.280 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.280 INFO:tasks.workunit.client.0.smithi142.stdout: "choose_args": {} 2020-03-11T22:48:42.280 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.280 INFO:tasks.workunit.client.0.smithi142.stdout: "fsmap": { 2020-03-11T22:48:42.280 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:48:42.281 INFO:tasks.workunit.client.0.smithi142.stdout: "default_fscid": -1, 2020-03-11T22:48:42.281 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": { 2020-03-11T22:48:42.281 INFO:tasks.workunit.client.0.smithi142.stdout: "compat": {}, 2020-03-11T22:48:42.281 INFO:tasks.workunit.client.0.smithi142.stdout: "ro_compat": {}, 2020-03-11T22:48:42.282 INFO:tasks.workunit.client.0.smithi142.stdout: "incompat": { 2020-03-11T22:48:42.282 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_1": "base v0.20", 2020-03-11T22:48:42.282 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:48:42.282 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:48:42.283 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:48:42.283 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:48:42.283 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:48:42.283 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_8": "no anchor table", 2020-03-11T22:48:42.283 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_9": "file layout v2", 2020-03-11T22:48:42.283 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:48:42.284 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.284 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.284 INFO:tasks.workunit.client.0.smithi142.stdout: "feature_flags": { 2020-03-11T22:48:42.284 INFO:tasks.workunit.client.0.smithi142.stdout: "enable_multiple": false, 2020-03-11T22:48:42.284 INFO:tasks.workunit.client.0.smithi142.stdout: "ever_enabled_multiple": false 2020-03-11T22:48:42.284 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.285 INFO:tasks.workunit.client.0.smithi142.stdout: "standbys": [], 2020-03-11T22:48:42.285 INFO:tasks.workunit.client.0.smithi142.stdout: "filesystems": [] 2020-03-11T22:48:42.285 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.285 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:48:42.285 INFO:tasks.workunit.client.0.smithi142.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:48:42.286 INFO:tasks.workunit.client.0.smithi142.stdout: "auth": { 2020-03-11T22:48:42.286 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:48:42.286 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 5, 2020-03-11T22:48:42.286 INFO:tasks.workunit.client.0.smithi142.stdout: "num_secrets": 3 2020-03-11T22:48:42.287 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.287 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:48:42.287 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_active": 1, 2020-03-11T22:48:42.287 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_unknown": 0, 2020-03-11T22:48:42.288 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osd": 3, 2020-03-11T22:48:42.288 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_sum": { 2020-03-11T22:48:42.288 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:48:42.288 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48430, 2020-03-11T22:48:42.288 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 101, 2020-03-11T22:48:42.288 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 1, 2020-03-11T22:48:42.289 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 303, 2020-03-11T22:48:42.289 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:48:42.289 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:48:42.289 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 101, 2020-03-11T22:48:42.289 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:48:42.289 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:48:42.290 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 101, 2020-03-11T22:48:42.290 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:48:42.290 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:48:42.290 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:48:42.290 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:48:42.290 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:48:42.291 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:48:42.291 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:48:42.291 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:48:42.291 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:48:42.292 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:48:42.292 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:48:42.292 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:48:42.292 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:48:42.293 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:48:42.293 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:48:42.293 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:48:42.293 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:48:42.293 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:48:42.294 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:48:42.294 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:48:42.294 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:48:42.294 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:48:42.295 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:48:42.295 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:48:42.295 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:48:42.295 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:48:42.296 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:48:42.296 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:48:42.296 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:48:42.297 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:48:42.297 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.297 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:48:42.298 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:48:42.298 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:48:42.298 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:48:42.298 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 0, 2020-03-11T22:48:42.298 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 0, 2020-03-11T22:48:42.299 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:42.299 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:42.299 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:42.300 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:48:42.300 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:48:42.300 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.300 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 18, 2020-03-11T22:48:42.301 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 18, 2020-03-11T22:48:42.301 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 3, 2020-03-11T22:48:42.301 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 3, 2020-03-11T22:48:42.301 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 0 2020-03-11T22:48:42.302 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.302 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum": { 2020-03-11T22:48:42.303 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:48:42.304 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:48:42.304 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:48:42.304 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:48:42.304 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:48:42.304 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:48:42.305 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:48:42.305 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3180992, 2020-03-11T22:48:42.305 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 35008, 2020-03-11T22:48:42.305 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 9, 2020-03-11T22:48:42.305 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145718, 2020-03-11T22:48:42.305 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311391808, 2020-03-11T22:48:42.306 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:48:42.306 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:48:42.306 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318865211392, 2020-03-11T22:48:42.306 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:48:42.307 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 35848192, 2020-03-11T22:48:42.307 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 1555460, 2020-03-11T22:48:42.307 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:42.307 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:42.307 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:42.307 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 9486, 2020-03-11T22:48:42.308 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221215986 2020-03-11T22:48:42.308 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.308 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:48:42.308 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:48:42.308 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:48:42.308 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:48:42.309 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:48:42.309 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:48:42.309 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:48:42.309 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.309 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:48:42.309 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:48:42.310 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:48:42.310 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:48:42.310 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:48:42.310 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.310 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:48:42.310 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:48:42.311 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.311 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_sum_by_class": { 2020-03-11T22:48:42.311 INFO:tasks.workunit.client.0.smithi142.stdout: "hdd": { 2020-03-11T22:48:42.311 INFO:tasks.workunit.client.0.smithi142.stdout: "up_from": 0, 2020-03-11T22:48:42.311 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 0, 2020-03-11T22:48:42.312 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pgs": 3, 2020-03-11T22:48:42.312 INFO:tasks.workunit.client.0.smithi142.stdout: "num_osds": 3, 2020-03-11T22:48:42.312 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_osds": 3, 2020-03-11T22:48:42.312 INFO:tasks.workunit.client.0.smithi142.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:48:42.312 INFO:tasks.workunit.client.0.smithi142.stdout: "kb": 314572800, 2020-03-11T22:48:42.312 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used": 3180992, 2020-03-11T22:48:42.313 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_data": 35008, 2020-03-11T22:48:42.313 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_omap": 9, 2020-03-11T22:48:42.313 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_used_meta": 3145718, 2020-03-11T22:48:42.313 INFO:tasks.workunit.client.0.smithi142.stdout: "kb_avail": 311391808, 2020-03-11T22:48:42.313 INFO:tasks.workunit.client.0.smithi142.stdout: "statfs": { 2020-03-11T22:48:42.314 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 322122547200, 2020-03-11T22:48:42.314 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 318865211392, 2020-03-11T22:48:42.314 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 3221225472, 2020-03-11T22:48:42.314 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 35848192, 2020-03-11T22:48:42.314 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 1555460, 2020-03-11T22:48:42.314 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:42.314 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:42.315 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:42.315 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 9486, 2020-03-11T22:48:42.315 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 3221215986 2020-03-11T22:48:42.317 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.317 INFO:tasks.workunit.client.0.smithi142.stdout: "hb_peers": [], 2020-03-11T22:48:42.317 INFO:tasks.workunit.client.0.smithi142.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:48:42.318 INFO:tasks.workunit.client.0.smithi142.stdout: "num_snap_trimming": 0, 2020-03-11T22:48:42.318 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shards_repaired": 0, 2020-03-11T22:48:42.318 INFO:tasks.workunit.client.0.smithi142.stdout: "op_queue_age_hist": { 2020-03-11T22:48:42.318 INFO:tasks.workunit.client.0.smithi142.stdout: "histogram": [], 2020-03-11T22:48:42.318 INFO:tasks.workunit.client.0.smithi142.stdout: "upper_bound": 1 2020-03-11T22:48:42.318 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.319 INFO:tasks.workunit.client.0.smithi142.stdout: "perf_stat": { 2020-03-11T22:48:42.319 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ms": 0, 2020-03-11T22:48:42.319 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ms": 0, 2020-03-11T22:48:42.319 INFO:tasks.workunit.client.0.smithi142.stdout: "commit_latency_ns": 0, 2020-03-11T22:48:42.319 INFO:tasks.workunit.client.0.smithi142.stdout: "apply_latency_ns": 0 2020-03-11T22:48:42.320 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.320 INFO:tasks.workunit.client.0.smithi142.stdout: "alerts": [], 2020-03-11T22:48:42.320 INFO:tasks.workunit.client.0.smithi142.stdout: "network_ping_times": [] 2020-03-11T22:48:42.320 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.320 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.320 INFO:tasks.workunit.client.0.smithi142.stdout: "pool_stats": [ 2020-03-11T22:48:42.321 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.321 INFO:tasks.workunit.client.0.smithi142.stdout: "poolid": 1, 2020-03-11T22:48:42.321 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg": 1, 2020-03-11T22:48:42.321 INFO:tasks.workunit.client.0.smithi142.stdout: "stat_sum": { 2020-03-11T22:48:42.321 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes": 48430, 2020-03-11T22:48:42.321 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects": 101, 2020-03-11T22:48:42.322 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_clones": 1, 2020-03-11T22:48:42.322 INFO:tasks.workunit.client.0.smithi142.stdout: "num_object_copies": 303, 2020-03-11T22:48:42.322 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:48:42.322 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_missing": 0, 2020-03-11T22:48:42.322 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_degraded": 101, 2020-03-11T22:48:42.322 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_misplaced": 0, 2020-03-11T22:48:42.323 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_unfound": 0, 2020-03-11T22:48:42.323 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_dirty": 101, 2020-03-11T22:48:42.323 INFO:tasks.workunit.client.0.smithi142.stdout: "num_whiteouts": 0, 2020-03-11T22:48:42.323 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read": 0, 2020-03-11T22:48:42.323 INFO:tasks.workunit.client.0.smithi142.stdout: "num_read_kb": 0, 2020-03-11T22:48:42.323 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write": 101, 2020-03-11T22:48:42.324 INFO:tasks.workunit.client.0.smithi142.stdout: "num_write_kb": 101, 2020-03-11T22:48:42.324 INFO:tasks.workunit.client.0.smithi142.stdout: "num_scrub_errors": 0, 2020-03-11T22:48:42.324 INFO:tasks.workunit.client.0.smithi142.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:48:42.324 INFO:tasks.workunit.client.0.smithi142.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:48:42.324 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_recovered": 0, 2020-03-11T22:48:42.324 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_recovered": 0, 2020-03-11T22:48:42.325 INFO:tasks.workunit.client.0.smithi142.stdout: "num_keys_recovered": 0, 2020-03-11T22:48:42.325 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_omap": 0, 2020-03-11T22:48:42.325 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:48:42.325 INFO:tasks.workunit.client.0.smithi142.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:48:42.325 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush": 0, 2020-03-11T22:48:42.325 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_kb": 0, 2020-03-11T22:48:42.326 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict": 0, 2020-03-11T22:48:42.326 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_kb": 0, 2020-03-11T22:48:42.326 INFO:tasks.workunit.client.0.smithi142.stdout: "num_promote": 0, 2020-03-11T22:48:42.326 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_high": 0, 2020-03-11T22:48:42.326 INFO:tasks.workunit.client.0.smithi142.stdout: "num_flush_mode_low": 0, 2020-03-11T22:48:42.326 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_some": 0, 2020-03-11T22:48:42.327 INFO:tasks.workunit.client.0.smithi142.stdout: "num_evict_mode_full": 0, 2020-03-11T22:48:42.327 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_pinned": 0, 2020-03-11T22:48:42.327 INFO:tasks.workunit.client.0.smithi142.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:48:42.327 INFO:tasks.workunit.client.0.smithi142.stdout: "num_large_omap_objects": 0, 2020-03-11T22:48:42.327 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_manifest": 0, 2020-03-11T22:48:42.327 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_bytes": 0, 2020-03-11T22:48:42.328 INFO:tasks.workunit.client.0.smithi142.stdout: "num_omap_keys": 0, 2020-03-11T22:48:42.328 INFO:tasks.workunit.client.0.smithi142.stdout: "num_objects_repaired": 0 2020-03-11T22:48:42.328 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.328 INFO:tasks.workunit.client.0.smithi142.stdout: "store_stats": { 2020-03-11T22:48:42.328 INFO:tasks.workunit.client.0.smithi142.stdout: "total": 0, 2020-03-11T22:48:42.328 INFO:tasks.workunit.client.0.smithi142.stdout: "available": 0, 2020-03-11T22:48:42.329 INFO:tasks.workunit.client.0.smithi142.stdout: "internally_reserved": 0, 2020-03-11T22:48:42.329 INFO:tasks.workunit.client.0.smithi142.stdout: "allocated": 19857408, 2020-03-11T22:48:42.329 INFO:tasks.workunit.client.0.smithi142.stdout: "data_stored": 1241088, 2020-03-11T22:48:42.329 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed": 0, 2020-03-11T22:48:42.329 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_allocated": 0, 2020-03-11T22:48:42.329 INFO:tasks.workunit.client.0.smithi142.stdout: "data_compressed_original": 0, 2020-03-11T22:48:42.330 INFO:tasks.workunit.client.0.smithi142.stdout: "omap_allocated": 0, 2020-03-11T22:48:42.330 INFO:tasks.workunit.client.0.smithi142.stdout: "internal_metadata": 0 2020-03-11T22:48:42.330 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.330 INFO:tasks.workunit.client.0.smithi142.stdout: "log_size": 18, 2020-03-11T22:48:42.330 INFO:tasks.workunit.client.0.smithi142.stdout: "ondisk_log_size": 18, 2020-03-11T22:48:42.330 INFO:tasks.workunit.client.0.smithi142.stdout: "up": 3, 2020-03-11T22:48:42.331 INFO:tasks.workunit.client.0.smithi142.stdout: "acting": 3, 2020-03-11T22:48:42.331 INFO:tasks.workunit.client.0.smithi142.stdout: "num_store_stats": 3 2020-03-11T22:48:42.331 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.331 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.331 INFO:tasks.workunit.client.0.smithi142.stdout: "osd_stats": [ 2020-03-11T22:48:42.331 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.331 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:42.332 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 176093659146 2020-03-11T22:48:42.332 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.332 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.332 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:42.332 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 42949672979 2020-03-11T22:48:42.332 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.333 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.333 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:42.333 INFO:tasks.workunit.client.0.smithi142.stdout: "seq": 188978561033 2020-03-11T22:48:42.333 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.333 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.333 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_state": [ 2020-03-11T22:48:42.334 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.334 INFO:tasks.workunit.client.0.smithi142.stdout: "state": "active+undersized+degraded", 2020-03-11T22:48:42.334 INFO:tasks.workunit.client.0.smithi142.stdout: "num": 1 2020-03-11T22:48:42.334 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.334 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.335 INFO:tasks.workunit.client.0.smithi142.stdout: "num_pg_by_osd": [ 2020-03-11T22:48:42.335 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.335 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2, 2020-03-11T22:48:42.335 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:48:42.335 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:48:42.335 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:42.335 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.336 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.336 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 2147483647, 2020-03-11T22:48:42.336 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:48:42.336 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:48:42.336 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:42.336 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.337 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.337 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 1, 2020-03-11T22:48:42.337 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 0, 2020-03-11T22:48:42.337 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 0, 2020-03-11T22:48:42.337 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:42.338 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.338 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.338 INFO:tasks.workunit.client.0.smithi142.stdout: "osd": 0, 2020-03-11T22:48:42.338 INFO:tasks.workunit.client.0.smithi142.stdout: "num_primary_pg": 1, 2020-03-11T22:48:42.338 INFO:tasks.workunit.client.0.smithi142.stdout: "num_acting_pg": 1, 2020-03-11T22:48:42.338 INFO:tasks.workunit.client.0.smithi142.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:48:42.339 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.339 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.339 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": [ 2020-03-11T22:48:42.339 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:48:42.339 INFO:tasks.workunit.client.0.smithi142.stdout: "pool": 1, 2020-03-11T22:48:42.339 INFO:tasks.workunit.client.0.smithi142.stdout: "purged_snaps": {} 2020-03-11T22:48:42.340 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.340 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:48:42.340 INFO:tasks.workunit.client.0.smithi142.stdout: "servicemap": { 2020-03-11T22:48:42.340 INFO:tasks.workunit.client.0.smithi142.stdout: "epoch": 1, 2020-03-11T22:48:42.340 INFO:tasks.workunit.client.0.smithi142.stdout: "modified": "2020-03-11T22:46:14.433718+0000", 2020-03-11T22:48:42.340 INFO:tasks.workunit.client.0.smithi142.stdout: "services": {} 2020-03-11T22:48:42.341 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:48:42.341 INFO:tasks.workunit.client.0.smithi142.stdout: "paxos": { 2020-03-11T22:48:42.341 INFO:tasks.workunit.client.0.smithi142.stdout: "first_committed": 1, 2020-03-11T22:48:42.341 INFO:tasks.workunit.client.0.smithi142.stdout: "last_committed": 428, 2020-03-11T22:48:42.341 INFO:tasks.workunit.client.0.smithi142.stdout: "last_pn": 0, 2020-03-11T22:48:42.341 INFO:tasks.workunit.client.0.smithi142.stdout: "accepted_pn": 0 2020-03-11T22:48:42.342 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:48:42.342 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:48:42.343 INFO:tasks.workunit.client.0.smithi142.stderr:report 1248230612 2020-03-11T22:48:42.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:48:42.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:48:42.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:42.344 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery 2020-03-11T22:48:42.479 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:42.480 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:48:07.496196+0000 39'102 45:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:48:42.480 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:48:42.480 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:42.481 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:48:42.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:48:42.490 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:48:42.491 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:48:42.491 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:48:42.491 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:42.491 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:42.491 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:48:42.491 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.0.asok 2020-03-11T22:48:42.491 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:48:42.492 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:48:42.582 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:48:42.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:48:42.590 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:48:42.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:48:42.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:48:42.591 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:42.591 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:42.591 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:48:42.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.2.asok 2020-03-11T22:48:42.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:48:42.592 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:48:42.681 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:48:42.688 INFO:tasks.workunit.client.0.smithi142.stdout:revive divergent 1 2020-03-11T22:48:42.688 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:48:42.689 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:43.008 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:43.008 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:48:07.496196+0000 39'102 45:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:48:43.009 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:48:43.009 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:43.010 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:48:43.018 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:48:43.637 INFO:tasks.workunit.client.0.smithi142.stderr:noup is set 2020-03-11T22:48:43.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:48:43.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:48:43.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:48:43.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:48:43.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:48:43.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:48:43.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:48:43.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:48:43.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:48:43.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:48:43.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:48:43.651 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:48:43.651 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:48:43.651 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:48:43.651 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:48:43.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:48:43.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:48:43.652 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:48:43.652 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:48:43.652 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:43.652 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:48:43.652 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:48:43.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:48:43.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:48:43.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:48:43.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:48:43.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:48:43.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:48:43.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:43.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:48:43.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:48:43.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:48:43.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:48:43.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:48:43.654 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:48:43.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:48:43.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:48:43.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:48:43.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:48:43.655 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:48:43.674 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:43.670+0000 7f65b05d1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:43.676 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:43.672+0000 7f65b05d1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:43.678 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:43.674+0000 7f65b05d1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:48:43.994 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:48:43.994 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:48:44.193 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:44.189+0000 7f65b05d1ec0 -1 Falling back to public interface 2020-03-11T22:48:44.986 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:48:44.982+0000 7f65b05d1ec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:48:48.995 INFO:tasks.workunit.client.0.smithi142.stdout:delay recovery divergent 2020-03-11T22:48:48.995 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:48:48.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:49.327 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:49.327 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:48:07.496196+0000 39'102 45:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:48:49.327 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:48:49.327 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:49.329 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:48:49.338 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:48:49.338 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:48:49.338 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:48:49.339 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:49.339 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:49.339 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:48:49.339 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17403/ceph-osd.1.asok 2020-03-11T22:48:49.339 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:48:49.339 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17403/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:48:49.429 INFO:tasks.workunit.client.0.smithi142.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:48:49.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:48:50.109 INFO:tasks.workunit.client.0.smithi142.stderr:noup is unset 2020-03-11T22:48:50.117 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:48:50.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:50.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:48:50.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:50.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:50.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:50.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:50.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:50.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:48:50.119 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:48:50.488 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 41 up_thru 44 down_at 40 last_clean_interval [35,39) [v2:127.0.0.1:6802/50308,v1:127.0.0.1:6803/50308] [v2:127.0.0.1:6804/50308,v1:127.0.0.1:6805/50308] exists,up 36e0b51c-dec2-42db-b07e-d9cf10df314e 2020-03-11T22:48:50.488 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:48:50.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:50.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:50.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:50.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:48:50.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:50.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:48:50.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:50.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:50.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:50.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:50.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:50.491 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:48:50.851 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/51936,v1:127.0.0.1:6819/51936] [v2:127.0.0.1:6820/51936,v1:127.0.0.1:6821/51936] exists,up 11c7e564-1ae2-4288-8725-ccd9a0814a84 2020-03-11T22:48:50.851 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:48:50.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:50.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:50.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:50.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:48:50.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:48:50.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:48:50.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:48:50.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:48:50.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:48:50.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:48:50.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:48:50.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:48:51.197 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 44 up_thru 0 down_at 43 last_clean_interval [36,39) [v2:127.0.0.1:6810/50540,v1:127.0.0.1:6811/50540] [v2:127.0.0.1:6812/50540,v1:127.0.0.1:6813/50540] exists,up 2c7e5aef-f703-4a53-8c31-d0d8c932abf4 2020-03-11T22:48:51.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:48:51.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:48:51.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:48:51.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:51.549 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:51.550 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 peering 2020-03-11T22:48:49.986455+0000 29'104 49:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:48:51.550 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:48:51.550 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:51.551 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:48:51.559 INFO:tasks.workunit.client.0.smithi142.stdout:wait for peering 2020-03-11T22:48:51.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:48:51.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:48:51.881 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:51.881 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 peering 2020-03-11T22:48:49.986455+0000 29'104 49:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:48:51.882 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:48:51.882 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:51.883 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:48:51.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:49:03.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:49:03.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:49:03.528 INFO:tasks.workunit.client.0.smithi142.stdout:killing divergent 1 2020-03-11T22:49:03.860 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:03.860 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:48:50.243131+0000 39'102 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:49:03.860 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:49:03.860 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:03.861 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:49:03.869 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:49:03.870 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:03.870 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:03.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:03.870 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:03.871 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:03.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:03.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:49:03.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:49:03.976 INFO:tasks.workunit.client.0.smithi142.stdout:reviving divergent 1 2020-03-11T22:49:04.298 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:04.298 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:48:50.243131+0000 39'102 50:129 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:49:04.298 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:49:04.298 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:04.299 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:49:04.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:49:04.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:49:04.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:04.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:49:04.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:04.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:49:04.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:49:04.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:04.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:04.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:04.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:49:04.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:49:04.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:04.312 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:04.313 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:49:04.313 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:04.313 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:04.313 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:04.313 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:04.313 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:04.313 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:49:04.314 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:49:04.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' 2020-03-11T22:49:04.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:04.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:49:04.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:49:04.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:04.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:04.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:04.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:04.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:04.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:49:04.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:49:04.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=e4d81029-c251-4939-a03a-37f830550d7a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:04.315 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:49:04.316 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:49:04.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:49:04.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:04.317 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:04.317 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:04.336 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:04.334+0000 7fb112bd4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:04.338 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:04.335+0000 7fb112bd4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:04.340 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:04.337+0000 7fb112bd4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:04.668 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:49:04.668 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:04.668 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:49:04.668 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:04.669 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:04.855 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:04.852+0000 7fb112bd4ec0 -1 Falling back to public interface 2020-03-11T22:49:05.010 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/51936,v1:127.0.0.1:6819/51936] [v2:127.0.0.1:6820/51936,v1:127.0.0.1:6821/51936] exists,up 11c7e564-1ae2-4288-8725-ccd9a0814a84 2020-03-11T22:49:05.011 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:05.011 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:05.011 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:05.011 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:49:05.635 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:05.633+0000 7fb112bd4ec0 -1 osd.1 50 log_to_monitors {default=true} 2020-03-11T22:49:25.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:49:25.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:49:25.012 INFO:tasks.workunit.client.0.smithi142.stdout:allowing recovery 2020-03-11T22:49:25.346 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:25.346 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:49:06.858909+0000 50'103 53:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:49:25.346 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:49:25.346 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:25.348 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:49:25.356 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:49:25.356 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:49:25.476 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:49:25.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:49:25.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:49:25.604 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:49:25.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:49:25.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:49:25.733 INFO:tasks.workunit.client.0.smithi142.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:49:25.741 INFO:tasks.workunit.client.0.smithi142.stdout:reading divergent objects 2020-03-11T22:49:25.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:49:25.741 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:49:26.062 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:49:26.062 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:49:06.858909+0000 50'103 53:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:46:36.308156+0000 0'0 2020-03-11T22:46:36.308156+0000 0 2020-03-11T22:49:26.062 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:49:26.062 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:49:26.063 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:49:26.072 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:49:26.073 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:49:26.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:49:26.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:49:26.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:49:26.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:49:26.153 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:49:26.153 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:49:26.182 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:49:26.182 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:49:26.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:49:26.211 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:49:26.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:49:26.216 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:46:57.100894+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.216 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:46:57.100894+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.216 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:49:26.216 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:49:26.216 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:49:26.216 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:49:26.217 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:47:03.704033+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.217 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:47:03.704033+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.217 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:49:26.217 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:49:26.217 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:49:26.217 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:49:26.217 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2020-03-11T22:49:26.218 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-03-11T22:47:03.704033+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.218 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:49:26.218 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-03-11T22:47:02.539941+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:47:03.541263+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.218 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-03-11T22:47:02.539941+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.219 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:47:03.541263+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.219 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:49:26.219 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2020-03-11T22:49:26.219 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:49:26.219 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:49:26.219 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2020-03-11T22:49:26.220 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-03-11T22:47:03.541263+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:49:26.220 INFO:tasks.workunit.client.0.smithi142.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:48:50.206+0000 7f658e78c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:49:26.220 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:49:26.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:49:26.222 INFO:tasks.workunit.client.0.smithi142.stdout:success 2020-03-11T22:49:26.222 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:49:26.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:49:26.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:49:26.224 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:49:26.669 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test' does not exist 2020-03-11T22:49:26.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:49:26.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:26.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:26.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:26.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:26.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:26.786 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:26.786 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:49:26.786 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:49:26.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:26.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:49:26.787 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:26.787 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:26.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:26.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:26.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:26.791 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:26.791 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:26.792 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:26.792 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:26.793 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:26.793 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:26.793 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:26.795 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:26.795 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:26.795 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:26.796 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:26.796 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:26.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:26.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:49:26.810 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:26.811 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:26.811 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:49:26.811 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:49:26.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:26.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:26.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:49:26.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:49:26.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:49:26.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:49:26.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:49:26.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:26.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:26.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:26.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:26.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:26.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:26.816 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:26.817 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:26.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:26.817 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:26.817 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:26.817 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:26.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:26.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:26.818 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:26.818 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:26.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:26.819 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:49:26.820 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:49:26.821 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:26.821 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:26.821 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17403 2020-03-11T22:49:26.821 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17403 2020-03-11T22:49:26.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:26.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:26.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:49:26.823 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:49:26.824 INFO:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:49:26.862 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:49:26.862 INFO:teuthology.orchestra.run.smithi142:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:49:26.920 INFO:tasks.workunit.client.0.smithi142.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:49:26.928 INFO:tasks.workunit.client.0.smithi142.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:49:26.928 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:49:26.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:49:26.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:49:26.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:49:26.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:49:26.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:49:26.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:49:26.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:49:26.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:49:26.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:49:26.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:49:26.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:49:26.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:49:26.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:49:26.930 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:49:26.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none ' 2020-03-11T22:49:26.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:26.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:49:26.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:49:26.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:49:26.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:49:26.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:49:26.932 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:49:26.932 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:49:26.932 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:49:26.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:49:26.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:49:26.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:49:26.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:49:26.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:49:26.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:49:26.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:26.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:49:26.935 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:26.935 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:26.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:26.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:26.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:26.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:26.936 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:26.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:26.936 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:26.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:26.937 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:26.937 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:26.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:26.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:26.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:26.939 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:26.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:26.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:26.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:49:26.939 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:26.940 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:26.940 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:26.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52864 2020-03-11T22:49:26.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:26.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:26.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:49:26.942 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:49:26.943 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:26.943 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:26.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52864 2020-03-11T22:49:26.943 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:49:26.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:49:26.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:49:26.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:49:26.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:49:26.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:49:26.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:49:26.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:49:26.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:49:26.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:49:26.944 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:49:26.945 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:49:26.945 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:49:26.945 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:49:27.202 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:49:27.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:27.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:27.203 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:27.203 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:27.203 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:27.203 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:27.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:49:27.361 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:49:27.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:49:27.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:27.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:27.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:49:27.363 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:49:27.364 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:27.364 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:27.364 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:27.364 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:27.364 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:27.364 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:27.364 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52864/ceph-mon.a.asok 2020-03-11T22:49:27.365 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:27.365 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52864/ceph-mon.a.asok config get fsid 2020-03-11T22:49:27.448 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:49:27.449 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:27.449 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:27.449 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:49:27.449 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:49:27.449 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:27.449 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:27.450 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:27.450 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:27.450 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:27.450 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:27.450 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52864/ceph-mon.a.asok 2020-03-11T22:49:27.450 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:27.450 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52864/ceph-mon.a.asok config get mon_host 2020-03-11T22:49:27.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:49:27.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:49:27.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:49:27.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:49:27.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:49:27.536 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:49:27.536 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:49:28.403 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:49:28.403 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:28.403 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:28.403 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:28.404 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:28.404 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:28.404 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:28.404 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:28.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:28.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:49:28.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:28.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:28.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:49:28.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:28.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:49:28.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:28.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:28.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:28.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:28.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:49:28.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:49:28.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:28.430 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:28.430 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:28.430 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:28.430 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:28.430 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:28.431 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:28.431 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:28.431 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:28.431 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:28.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:28.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:28.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:28.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:28.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:28.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:28.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:28.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:28.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:28.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:28.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:28.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:49:28.433 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:28.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3ab0baa4-f04c-4e67-af07-4ecec8f3ce26 2020-03-11T22:49:28.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 3ab0baa4-f04c-4e67-af07-4ecec8f3ce26' 2020-03-11T22:49:28.433 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:28.434 INFO:tasks.workunit.client.0.smithi142.stdout:add osd0 3ab0baa4-f04c-4e67-af07-4ecec8f3ce26 2020-03-11T22:49:28.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD4amled7lyGhAAWQER339Mb1nzOjCaMHpARA== 2020-03-11T22:49:28.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD4amled7lyGhAAWQER339Mb1nzOjCaMHpARA=="}' 2020-03-11T22:49:28.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3ab0baa4-f04c-4e67-af07-4ecec8f3ce26 -i td/ec-error-rollforward/0/new.json 2020-03-11T22:49:28.785 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:28.794 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:49:28.795 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD4amled7lyGhAAWQER339Mb1nzOjCaMHpARA== --osd-uuid 3ab0baa4-f04c-4e67-af07-4ecec8f3ce26 2020-03-11T22:49:28.815 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:28.812+0000 7fda8786fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:28.816 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:28.814+0000 7fda8786fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:28.818 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:28.815+0000 7fda8786fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:28.819 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:28.817+0000 7fda8786fec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:49:28.820 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:28.817+0000 7fda8786fec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:49:28.820 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:28.817+0000 7fda8786fec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:49:28.860 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:28.858+0000 7fda8786fec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:49:31.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:49:31.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:31.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:49:31.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:31.864 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd0 key to auth repository 2020-03-11T22:49:32.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:49:32.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:32.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:32.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:32.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:32.305 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:49:32.323 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:32.319+0000 7f5462af0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:32.324 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:32.321+0000 7f5462af0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:32.325 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:32.322+0000 7f5462af0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:32.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:49:32.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:32.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:49:32.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:32.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:32.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:32.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:32.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:32.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:32.646 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:32.841 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:32.837+0000 7f5462af0ec0 -1 Falling back to public interface 2020-03-11T22:49:32.995 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:33.575 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:33.571+0000 7f5462af0ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:49:33.996 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:33.997 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:33.997 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:33.997 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:33.997 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:33.997 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:49:34.407 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53293,v1:127.0.0.1:6803/53293] [v2:127.0.0.1:6804/53293,v1:127.0.0.1:6805/53293] exists,up 3ab0baa4-f04c-4e67-af07-4ecec8f3ce26 2020-03-11T22:49:34.407 INFO:tasks.workunit.client.0.smithi142.stdout:add osd1 bc151370-f483-4ecc-a117-03a8fb59f51c 2020-03-11T22:49:34.407 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:34.407 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:34.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:34.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:49:34.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:34.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:34.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:49:34.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:34.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:49:34.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:34.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:34.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:34.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:34.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:49:34.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:49:34.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:34.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:34.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:34.410 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:34.410 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:34.410 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:34.410 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:34.410 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:34.411 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:34.411 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:34.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:34.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:34.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:34.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:34.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:34.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:34.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:34.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:34.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:34.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:34.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:34.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:49:34.413 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:34.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bc151370-f483-4ecc-a117-03a8fb59f51c 2020-03-11T22:49:34.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 bc151370-f483-4ecc-a117-03a8fb59f51c' 2020-03-11T22:49:34.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:34.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD+amlepYHPGBAAWelxFUAQJ1gumkZW2eJwAA== 2020-03-11T22:49:34.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD+amlepYHPGBAAWelxFUAQJ1gumkZW2eJwAA=="}' 2020-03-11T22:49:34.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bc151370-f483-4ecc-a117-03a8fb59f51c -i td/ec-error-rollforward/1/new.json 2020-03-11T22:49:34.829 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:49:34.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:49:34.840 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+amlepYHPGBAAWelxFUAQJ1gumkZW2eJwAA== --osd-uuid bc151370-f483-4ecc-a117-03a8fb59f51c 2020-03-11T22:49:34.860 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:34.856+0000 7faee9cd3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:34.862 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:34.858+0000 7faee9cd3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:34.864 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:34.860+0000 7faee9cd3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:34.865 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:34.861+0000 7faee9cd3ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:49:34.865 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:34.861+0000 7faee9cd3ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:49:34.865 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:34.861+0000 7faee9cd3ec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:49:34.941 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:34.938+0000 7faee9cd3ec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:49:37.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:49:37.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:37.892 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd1 key to auth repository 2020-03-11T22:49:37.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:49:37.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:38.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:49:38.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:38.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:38.345 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:38.345 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:38.345 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:49:38.362 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:38.358+0000 7f445f2feec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:38.364 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:38.360+0000 7f445f2feec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:38.365 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:38.361+0000 7f445f2feec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:38.684 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:49:38.684 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:38.685 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:49:38.685 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:38.685 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:38.685 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:38.685 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:38.685 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:38.686 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:38.686 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:38.880 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:38.876+0000 7f445f2feec0 -1 Falling back to public interface 2020-03-11T22:49:39.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:39.614 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:39.610+0000 7f445f2feec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:49:40.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:40.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:40.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:40.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:40.063 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:40.064 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:49:40.411 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53648,v1:127.0.0.1:6811/53648] [v2:127.0.0.1:6812/53648,v1:127.0.0.1:6813/53648] exists,up bc151370-f483-4ecc-a117-03a8fb59f51c 2020-03-11T22:49:40.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:40.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:40.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:40.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:49:40.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:40.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:40.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:49:40.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:40.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:49:40.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:40.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:40.414 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:40.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:40.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:40.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:40.414 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:40.414 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:40.415 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:40.415 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:40.415 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:40.415 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:40.415 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:40.415 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:40.415 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:40.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:40.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:40.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:40.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:40.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:40.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:40.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:49:40.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:40.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7253251b-c258-4839-bfee-81f99a458764 2020-03-11T22:49:40.417 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 7253251b-c258-4839-bfee-81f99a458764' 2020-03-11T22:49:40.418 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:40.418 INFO:tasks.workunit.client.0.smithi142.stdout:add osd2 7253251b-c258-4839-bfee-81f99a458764 2020-03-11T22:49:40.425 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAEa2le/FQtGRAAdtqU27T6a8QRVfQ81qGwTw== 2020-03-11T22:49:40.425 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAEa2le/FQtGRAAdtqU27T6a8QRVfQ81qGwTw=="}' 2020-03-11T22:49:40.425 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7253251b-c258-4839-bfee-81f99a458764 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:49:40.860 INFO:tasks.workunit.client.0.smithi142.stdout:2 2020-03-11T22:49:40.871 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:49:40.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAEa2le/FQtGRAAdtqU27T6a8QRVfQ81qGwTw== --osd-uuid 7253251b-c258-4839-bfee-81f99a458764 2020-03-11T22:49:40.891 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:40.887+0000 7f44ebc79ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:40.892 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:40.888+0000 7f44ebc79ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:40.894 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:40.890+0000 7f44ebc79ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:40.895 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:40.891+0000 7f44ebc79ec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:49:40.895 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:40.891+0000 7f44ebc79ec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:49:40.895 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:40.891+0000 7f44ebc79ec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:49:40.971 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:40.968+0000 7f44ebc79ec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:49:43.906 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:49:43.906 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:43.907 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd2 key to auth repository 2020-03-11T22:49:43.907 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:49:43.907 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:44.358 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:49:44.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:49:44.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:44.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:44.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:44.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:44.376 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:44.373+0000 7f3365219ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:44.378 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:44.374+0000 7f3365219ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:44.380 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:44.376+0000 7f3365219ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:44.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:49:44.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:44.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:49:44.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:44.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:44.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:44.702 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:44.702 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:44.702 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:44.702 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:44.896 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:44.892+0000 7f3365219ec0 -1 Falling back to public interface 2020-03-11T22:49:45.043 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:45.645 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:45.641+0000 7f3365219ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:49:46.044 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:46.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:46.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:46.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:46.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:46.045 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54003,v1:127.0.0.1:6819/54003] [v2:127.0.0.1:6820/54003,v1:127.0.0.1:6821/54003] exists,up 7253251b-c258-4839-bfee-81f99a458764 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:46.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:46.433 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:49:46.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:46.435 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:46.435 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:46.435 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:46.435 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:46.435 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:46.436 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:46.436 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:46.436 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:46.436 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:46.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:46.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:46.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:46.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:46.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:46.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:46.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:46.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:46.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:46.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:46.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:46.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:49:46.439 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:46.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=94834095-756d-4220-adbe-150c3f4aae20 2020-03-11T22:49:46.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 94834095-756d-4220-adbe-150c3f4aae20' 2020-03-11T22:49:46.439 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:46.439 INFO:tasks.workunit.client.0.smithi142.stdout:add osd3 94834095-756d-4220-adbe-150c3f4aae20 2020-03-11T22:49:46.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAKa2les6V7GhAAx/K59JrZiIaIKBcujXrhcw== 2020-03-11T22:49:46.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAKa2les6V7GhAAx/K59JrZiIaIKBcujXrhcw=="}' 2020-03-11T22:49:46.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 94834095-756d-4220-adbe-150c3f4aae20 -i td/ec-error-rollforward/3/new.json 2020-03-11T22:49:46.957 INFO:tasks.workunit.client.0.smithi142.stdout:3 2020-03-11T22:49:46.966 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:49:46.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAKa2les6V7GhAAx/K59JrZiIaIKBcujXrhcw== --osd-uuid 94834095-756d-4220-adbe-150c3f4aae20 2020-03-11T22:49:46.987 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:46.984+0000 7f773ef8eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:46.989 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:46.985+0000 7f773ef8eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:46.990 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:46.987+0000 7f773ef8eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:46.991 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:46.987+0000 7f773ef8eec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:49:46.991 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:46.987+0000 7f773ef8eec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:49:46.991 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:46.987+0000 7f773ef8eec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:49:47.068 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:47.065+0000 7f773ef8eec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:49:50.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:49:50.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:50.070 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:49:50.070 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:50.070 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd3 key to auth repository 2020-03-11T22:49:50.521 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.3 2020-03-11T22:49:50.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:49:50.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:50.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:50.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:50.523 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:50.539 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:50.536+0000 7f14f996bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:50.540 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:50.538+0000 7f14f996bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:50.542 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:50.539+0000 7f14f996bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:50.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:49:50.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:50.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:49:50.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:50.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:50.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:50.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:50.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:50.864 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:50.865 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:51.061 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:51.058+0000 7f14f996bec0 -1 Falling back to public interface 2020-03-11T22:49:51.205 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:51.822 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:51.819+0000 7f14f996bec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:49:52.206 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:52.207 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:52.207 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:52.207 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:52.207 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:52.207 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:49:52.575 INFO:tasks.workunit.client.0.smithi142.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54358,v1:127.0.0.1:6827/54358] [v2:127.0.0.1:6828/54358,v1:127.0.0.1:6829/54358] exists,up 94834095-756d-4220-adbe-150c3f4aae20 2020-03-11T22:49:52.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:52.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:52.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:52.576 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:49:53.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:49:53.836 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'ec' already exists 2020-03-11T22:49:53.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:49:55.536 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:49:55.537 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 54003 2020-03-11T22:49:55.537 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=54613 2020-03-11T22:49:55.538 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:49:55.538 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:49:56.539 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 54636' 2020-03-11T22:49:56.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:49:56.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 54637' 2020-03-11T22:49:56.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:49:56.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 54638' 2020-03-11T22:49:56.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:49:56.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:49:57.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:49:57.541 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:57.541 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:57.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:57.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:57.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:57.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:57.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 54613 54636 54637 54638 2020-03-11T22:49:57.651 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:49:57.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:49:57.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:57.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:57.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:49:57.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:57.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:49:57.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:57.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:57.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:57.656 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:57.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:49:57.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:49:57.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:57.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:57.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:57.657 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:57.657 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:57.658 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:57.658 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:57.658 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:57.658 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:57.658 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:57.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:57.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:57.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:57.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:57.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:57.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:57.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:57.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:57.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:57.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:49:57.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:49:57.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:57.660 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:49:57.660 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:49:57.661 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:49:57.661 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:57.662 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:57.662 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:57.675 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:57.672+0000 7f2ed5959ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:57.705 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:57.702+0000 7f2ed5959ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:57.707 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:57.704+0000 7f2ed5959ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:58.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:49:58.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:58.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:49:58.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:58.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:58.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:58.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:58.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:58.005 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:58.005 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:58.222 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:58.219+0000 7f2ed5959ec0 -1 Falling back to public interface 2020-03-11T22:49:58.353 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53293,v1:127.0.0.1:6803/53293] [v2:127.0.0.1:6804/53293,v1:127.0.0.1:6805/53293] exists,up 3ab0baa4-f04c-4e67-af07-4ecec8f3ce26 2020-03-11T22:49:58.353 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:49:58.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:58.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:58.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:58.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:49:58.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:58.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:58.354 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:49:58.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:58.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:49:58.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:58.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:58.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:58.356 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:58.356 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:49:58.356 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:49:58.356 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:58.356 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:58.357 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:58.357 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:58.357 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:58.357 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:58.357 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:58.358 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:58.358 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:58.358 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:58.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:58.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:58.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:58.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:58.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:58.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:58.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:58.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:58.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:58.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:49:58.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:49:58.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:58.360 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:49:58.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:49:58.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:58.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:58.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:58.369 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:58.366+0000 7fadb4391ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:58.371 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:58.368+0000 7fadb4391ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:58.373 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:58.370+0000 7fadb4391ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:58.700 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:49:58.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:58.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:49:58.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:58.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:58.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:58.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:58.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:58.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:58.712 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:58.892 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:58.889+0000 7fadb4391ec0 -1 Falling back to public interface 2020-03-11T22:49:59.080 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53648,v1:127.0.0.1:6811/53648] [v2:127.0.0.1:6812/53648,v1:127.0.0.1:6813/53648] exists,up bc151370-f483-4ecc-a117-03a8fb59f51c 2020-03-11T22:49:59.080 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:49:59.080 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:59.080 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:59.081 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:59.081 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:49:59.081 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:59.081 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:59.081 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:49:59.081 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:59.081 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:49:59.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:59.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:59.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:59.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:59.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:49:59.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:49:59.082 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:59.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:59.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:59.083 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:59.083 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:59.083 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:59.083 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:59.084 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:59.084 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:59.084 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:59.084 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:59.084 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:59.084 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:59.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:59.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:59.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:59.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:59.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:59.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:59.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:49:59.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:49:59.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:59.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:49:59.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:49:59.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:59.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:59.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:59.097 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.094+0000 7fac5cd5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:59.098 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.096+0000 7fac5cd5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:59.100 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.097+0000 7fac5cd5aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:59.135 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.132+0000 7f2ed5959ec0 -1 osd.0 25 log_to_monitors {default=true} 2020-03-11T22:49:59.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:49:59.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:59.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:49:59.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:59.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:59.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:59.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:59.439 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:59.440 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:59.440 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:49:59.620 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.617+0000 7fac5cd5aec0 -1 Falling back to public interface 2020-03-11T22:49:59.881 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54003,v1:127.0.0.1:6819/54003] [v2:127.0.0.1:6820/54003,v1:127.0.0.1:6821/54003] exists,up 7253251b-c258-4839-bfee-81f99a458764 2020-03-11T22:49:59.881 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.3 2020-03-11T22:49:59.881 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:59.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:59.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:59.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:49:59.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:49:59.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:49:59.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:49:59.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:49:59.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:49:59.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:49:59.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:59.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:59.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:59.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:49:59.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:59.884 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:59.885 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:49:59.885 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:59.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' 2020-03-11T22:49:59.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:59.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:49:59.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:49:59.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:59.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:59.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:59.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:49:59.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:49:59.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:49:59.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:49:59.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=02d11951-642a-4f15-9e33-fac115cf731a --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52864/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:59.887 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:49:59.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:49:59.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:49:59.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:49:59.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:49:59.887 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.834+0000 7f0e5246aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:59.913 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.907+0000 7f0e5246aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:59.915 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:49:59.909+0000 7f0e5246aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:00.035 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:00.024+0000 7fadb4391ec0 -1 osd.1 25 log_to_monitors {default=true} 2020-03-11T22:50:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:50:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:50:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:50:00.189 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:50:00.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:50:00.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:50:00.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:50:00.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:50:00.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:50:00.190 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:50:00.433 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:00.427+0000 7f0e5246aec0 -1 Falling back to public interface 2020-03-11T22:50:00.572 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:50:00.806 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:00.803+0000 7fac5cd5aec0 -1 osd.2 24 log_to_monitors {default=true} 2020-03-11T22:50:01.542 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:01.539+0000 7f0e5246aec0 -1 osd.3 25 log_to_monitors {default=true} 2020-03-11T22:50:01.574 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:50:01.574 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:50:01.574 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:50:01.575 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:50:01.575 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:50:01.575 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:50:01.938 INFO:tasks.workunit.client.0.smithi142.stdout:osd.3 up in weight 1 up_from 35 up_thru 0 down_at 28 last_clean_interval [19,25) [v2:127.0.0.1:6826/55157,v1:127.0.0.1:6827/55157] [v2:127.0.0.1:6828/55157,v1:127.0.0.1:6829/55157] exists,up 94834095-756d-4220-adbe-150c3f4aae20 2020-03-11T22:50:01.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:50:01.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:50:01.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:50:01.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:50:01.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:01.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:01.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:01.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:01.940 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:01.940 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:01.941 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:01.941 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:01.941 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:01.941 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:02.026 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:02.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:02.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:02.027 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:02.027 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:02.370 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:02.370 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:50:02.370 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:50:02.371 INFO:tasks.workunit.client.0.smithi142.stderr:3' 2020-03-11T22:50:02.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:02.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:02.371 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:02.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=115964116995 2020-03-11T22:50:02.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-115964116995' 2020-03-11T22:50:02.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:02.491 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:02.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018883 2020-03-11T22:50:02.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-115964116995 1-128849018883' 2020-03-11T22:50:02.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:02.612 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:02.732 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-03-11T22:50:02.733 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-115964116995 1-128849018883 2-141733920771' 2020-03-11T22:50:02.733 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:02.733 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:02.852 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:50:02.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-115964116995 1-128849018883 2-141733920771 3-150323855363' 2020-03-11T22:50:02.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:02.853 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-115964116995 2020-03-11T22:50:02.853 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:02.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:02.855 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-115964116995 2020-03-11T22:50:02.855 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:02.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=115964116995 2020-03-11T22:50:02.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 115964116995' 2020-03-11T22:50:02.856 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 115964116995 2020-03-11T22:50:02.857 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:03.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 115964116995 2020-03-11T22:50:03.198 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:04.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:04.199 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:04.553 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 115964116995 -lt 115964116995 2020-03-11T22:50:04.553 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:04.554 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-128849018883 2020-03-11T22:50:04.554 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:04.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:04.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-128849018883 2020-03-11T22:50:04.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:04.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=128849018883 2020-03-11T22:50:04.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 128849018883' 2020-03-11T22:50:04.557 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 128849018883 2020-03-11T22:50:04.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:04.921 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 128849018883 -lt 128849018883 2020-03-11T22:50:04.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:04.922 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-141733920771 2020-03-11T22:50:04.922 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:04.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:04.924 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-141733920771 2020-03-11T22:50:04.924 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:04.925 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=141733920771 2020-03-11T22:50:04.925 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2020-03-11T22:50:04.925 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 141733920771 2020-03-11T22:50:04.926 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:05.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 141733920771 -lt 141733920771 2020-03-11T22:50:05.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:05.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-150323855363 2020-03-11T22:50:05.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:05.262 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:05.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-150323855363 2020-03-11T22:50:05.263 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:05.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:50:05.264 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2020-03-11T22:50:05.264 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 150323855363 2020-03-11T22:50:05.265 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:05.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:50:05.603 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:05.604 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:05.604 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:06.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:50:06.041 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:06.042 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:06.043 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:06.043 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:06.043 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:06.043 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:06.043 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:06.368 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:50:06.369 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:06.369 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:06.369 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:06.812 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:50:06.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:06.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:06.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:50:06.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:50:06.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:50:06.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:50:06.814 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:50:06.814 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:50:06.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:50:06.815 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:50:06.815 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:50:06.920 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:50:06.920 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:50:06.921 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:50:06.921 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:50:06.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:50:06.922 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:50:06.922 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:50:06.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:50:06.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:50:06.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:50:06.925 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:50:06.925 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:50:06.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:50:06.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:50:06.932 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:50:06.932 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:06.932 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:50:06.932 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52864 2020-03-11T22:50:06.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:50:06.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:50:06.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:50:06.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:50:06.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:50:06.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:50:06.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:50:06.935 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:50:06.935 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:50:06.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:50:06.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:50:06.936 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:50:06.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:50:06.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:50:06.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:50:06.938 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:50:06.938 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:50:06.939 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:50:06.939 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:50:06.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:50:06.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:50:06.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:50:06.940 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:50:06.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:50:06.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:50:06.941 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:50:06.941 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:50:06.942 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:06.942 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52864 2020-03-11T22:50:06.942 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52864 2020-03-11T22:50:06.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:50:06.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:50:06.943 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:50:06.944 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:50:06.944 INFO:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:50:07.019 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:50:07.019 INFO:teuthology.orchestra.run.smithi142:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:50:07.071 INFO:tasks.workunit.client.0.smithi142.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:50:07.085 INFO:tasks.workunit.client.0.smithi142.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:50:07.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:50:07.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:50:07.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:50:07.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:50:07.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:50:07.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:50:07.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:50:07.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:50:07.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:50:07.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:50:07.086 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:50:07.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:50:07.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:50:07.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:50:07.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:50:07.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none ' 2020-03-11T22:50:07.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:50:07.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:50:07.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:50:07.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:50:07.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:50:07.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:50:07.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:50:07.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:50:07.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:50:07.089 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:50:07.089 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:50:07.089 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:50:07.089 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:50:07.089 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:50:07.089 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:50:07.089 INFO:tasks.workunit.client.0.smithi142.stderr:TEST_backfill_priority' 2020-03-11T22:50:07.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:50:07.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:50:07.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:50:07.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:50:07.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:50:07.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:50:07.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:50:07.091 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:50:07.091 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:50:07.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:50:07.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:50:07.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:50:07.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:50:07.091 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:50:07.092 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:50:07.092 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:50:07.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:50:07.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:50:07.096 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:50:07.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:50:07.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:50:07.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:50:07.096 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:50:07.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:50:07.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:50:07.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:50:07.097 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:50:07.097 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:07.097 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:07.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55987 2020-03-11T22:50:07.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:50:07.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:50:07.098 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:50:07.098 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:50:07.099 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:07.099 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:07.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.55987 2020-03-11T22:50:07.099 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:50:07.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:50:07.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:50:07.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:50:07.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:50:07.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:50:07.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:50:07.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:50:07.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:50:07.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:50:07.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:50:07.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:50:07.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:50:07.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:50:07.102 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:50:07.102 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:50:07.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:50:07.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:50:07.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:50:07.386 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:50:07.387 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:07.387 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:07.387 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:50:07.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:50:07.554 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:50:07.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:50:07.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:50:07.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:50:07.556 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:50:07.557 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:50:07.557 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:50:07.557 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:50:07.557 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:50:07.557 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:07.557 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:07.558 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:07.558 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-mon.a.asok 2020-03-11T22:50:07.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:50:07.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55987/ceph-mon.a.asok config get fsid 2020-03-11T22:50:07.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:50:07.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:50:07.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:50:07.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:07.643 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-mon.a.asok 2020-03-11T22:50:07.644 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:50:07.644 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55987/ceph-mon.a.asok config get mon_host 2020-03-11T22:50:07.728 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:50:07.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:50:07.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:50:07.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:50:07.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:50:07.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:50:07.729 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:50:08.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:50:08.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:50:08.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:50:08.605 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:50:08.605 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:08.605 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:08.606 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:50:08.606 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:50:08.606 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:50:08.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:50:08.627 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:50:08.628 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:50:08.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:50:08.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:50:08.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:50:08.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:50:08.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:50:08.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:50:08.633 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:50:08.633 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:50:08.633 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:50:08.633 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:50:08.633 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:50:08.633 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:50:08.633 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:50:08.634 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:50:08.634 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:50:08.634 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:50:08.634 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:50:08.634 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:50:08.634 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:50:08.634 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:50:08.635 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:08.635 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:08.635 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:50:08.635 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:50:08.635 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:50:08.635 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:50:08.635 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:50:08.636 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:50:08.636 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:50:08.636 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:50:08.636 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:50:08.636 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:50:08.636 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:50:08.636 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:50:08.637 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:50:08.637 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:50:08.637 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=32bec9a0-970e-467f-aedd-8e08a7ccfa41 2020-03-11T22:50:08.637 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 32bec9a0-970e-467f-aedd-8e08a7ccfa41' 2020-03-11T22:50:08.637 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:50:08.637 INFO:tasks.workunit.client.0.smithi142.stdout:add osd0 32bec9a0-970e-467f-aedd-8e08a7ccfa41 2020-03-11T22:50:08.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAga2le+jpvJhAAP936zXCFLhPogN43uefJBA== 2020-03-11T22:50:08.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAga2le+jpvJhAAP936zXCFLhPogN43uefJBA=="}' 2020-03-11T22:50:08.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 32bec9a0-970e-467f-aedd-8e08a7ccfa41 -i td/osd-backfill-prio/0/new.json 2020-03-11T22:50:08.992 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:50:09.001 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:50:09.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAga2le+jpvJhAAP936zXCFLhPogN43uefJBA== --osd-uuid 32bec9a0-970e-467f-aedd-8e08a7ccfa41 2020-03-11T22:50:09.027 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:09.024+0000 7f14f93d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:09.029 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:09.026+0000 7f14f93d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:09.030 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:09.027+0000 7f14f93d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:09.030 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:09.027+0000 7f14f93d9ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:50:09.031 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:09.027+0000 7f14f93d9ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:50:09.031 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:09.027+0000 7f14f93d9ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:50:09.069 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:09.066+0000 7f14f93d9ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:50:12.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:50:12.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:50:12.046 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd0 key to auth repository 2020-03-11T22:50:12.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:50:12.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:50:12.496 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:50:12.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:50:12.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:50:12.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:50:12.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:50:12.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:50:12.520 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:12.517+0000 7faeac6d6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:12.521 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:12.519+0000 7faeac6d6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:12.523 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:12.520+0000 7faeac6d6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:12.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:50:12.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:50:12.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:50:12.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:50:12.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:50:12.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:50:12.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:50:12.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:50:12.845 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:50:12.846 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:50:13.037 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:13.034+0000 7faeac6d6ec0 -1 Falling back to public interface 2020-03-11T22:50:13.197 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:50:13.768 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:13.765+0000 7faeac6d6ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:50:14.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:50:14.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:50:14.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:50:14.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:50:14.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:50:14.203 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:50:14.559 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56419,v1:127.0.0.1:6803/56419] [v2:127.0.0.1:6804/56419,v1:127.0.0.1:6805/56419] exists,up 32bec9a0-970e-467f-aedd-8e08a7ccfa41 2020-03-11T22:50:14.559 INFO:tasks.workunit.client.0.smithi142.stdout:add osd1 c9122465-cd9a-4e13-a00f-59c69e30466d 2020-03-11T22:50:14.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:50:14.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:50:14.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:50:14.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:50:14.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:50:14.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:50:14.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:50:14.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:50:14.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:50:14.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:50:14.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:50:14.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:50:14.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:50:14.562 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:50:14.563 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:50:14.563 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:50:14.563 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:50:14.563 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:50:14.563 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:50:14.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:50:14.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:50:14.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:50:14.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:50:14.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:50:14.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:50:14.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:50:14.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:50:14.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:50:14.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:50:14.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:50:14.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:50:14.565 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:50:14.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c9122465-cd9a-4e13-a00f-59c69e30466d 2020-03-11T22:50:14.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 c9122465-cd9a-4e13-a00f-59c69e30466d' 2020-03-11T22:50:14.565 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:50:14.575 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAma2leiKIcIhAAuJ9JE9zYeFafAdVLpRTUnA== 2020-03-11T22:50:14.575 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAma2leiKIcIhAAuJ9JE9zYeFafAdVLpRTUnA=="}' 2020-03-11T22:50:14.575 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c9122465-cd9a-4e13-a00f-59c69e30466d -i td/osd-backfill-prio/1/new.json 2020-03-11T22:50:15.056 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:50:15.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:50:15.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAma2leiKIcIhAAuJ9JE9zYeFafAdVLpRTUnA== --osd-uuid c9122465-cd9a-4e13-a00f-59c69e30466d 2020-03-11T22:50:15.092 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:15.089+0000 7f26aa7b4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:15.093 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:15.091+0000 7f26aa7b4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:15.095 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:15.092+0000 7f26aa7b4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:15.095 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:15.092+0000 7f26aa7b4ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:50:15.096 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:15.092+0000 7f26aa7b4ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:50:15.096 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:15.092+0000 7f26aa7b4ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:50:15.166 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:15.163+0000 7f26aa7b4ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:50:18.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:50:18.199 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:50:18.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:50:18.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:50:18.201 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd1 key to auth repository 2020-03-11T22:50:18.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:50:18.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:50:18.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:50:18.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:50:18.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:50:18.659 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:50:18.682 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:18.679+0000 7f9413047ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:18.684 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:18.681+0000 7f9413047ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:18.685 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:18.682+0000 7f9413047ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:50:19.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:50:19.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:50:19.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:50:19.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:50:19.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:50:19.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:50:19.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:50:19.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:50:19.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:50:19.003 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:50:19.200 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:19.198+0000 7f9413047ec0 -1 Falling back to public interface 2020-03-11T22:50:19.347 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:50:19.976 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:50:19.972+0000 7f9413047ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:50:20.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:50:20.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:50:20.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:50:20.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:50:20.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:50:20.350 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:50:20.714 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56782,v1:127.0.0.1:6811/56782] [v2:127.0.0.1:6812/56782,v1:127.0.0.1:6813/56782] exists,up c9122465-cd9a-4e13-a00f-59c69e30466d 2020-03-11T22:50:20.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:50:20.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:50:20.714 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:50:20.715 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:50:20.715 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:20.715 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:50:20.715 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:50:21.290 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test1' already exists 2020-03-11T22:50:21.297 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:22.299 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:50:23.110 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 size to 2 2020-03-11T22:50:23.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:23.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:50:23.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:50:23.599 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test2' already exists 2020-03-11T22:50:23.607 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:24.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:50:25.155 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 2 size to 2 2020-03-11T22:50:25.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:25.163 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:50:25.164 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:50:25.704 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test3' already exists 2020-03-11T22:50:25.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:26.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:50:27.309 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 3 size to 2 2020-03-11T22:50:27.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:50:32.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:50:32.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:32.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:32.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:32.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:32.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:32.320 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:32.320 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:32.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:32.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:32.320 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:32.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:32.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:32.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:32.766 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:32.767 INFO:tasks.workunit.client.0.smithi142.stderr:1' 2020-03-11T22:50:32.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:32.767 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:32.893 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:50:32.894 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:50:32.894 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:32.894 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:33.019 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:50:33.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:50:33.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:33.020 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:50:33.021 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:33.021 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:33.023 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:50:33.023 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:33.024 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:50:33.024 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:50:33.024 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:50:33.024 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:33.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:50:33.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:34.370 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:34.371 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:34.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:50:34.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:34.722 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:50:34.723 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:34.723 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:34.724 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:50:34.724 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:34.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:50:34.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:50:34.726 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:50:34.726 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:35.071 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:50:35.072 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:35.072 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:35.072 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:35.516 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:50:35.516 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:35.517 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:35.517 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:35.517 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:35.517 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:35.518 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:35.518 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:35.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:50:35.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:35.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:35.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:36.295 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:50:36.295 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:36.295 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:36.295 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:50:36.615 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:36.615 INFO:tasks.workunit.client.0.smithi142.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:27.760343+0000 0'0 27:17 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:50:25.467863+0000 0'0 2020-03-11T22:50:25.467863+0000 0 2020-03-11T22:50:36.615 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:23.132073+0000 0'0 26:28 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:50:21.034174+0000 0'0 2020-03-11T22:50:21.034174+0000 0 2020-03-11T22:50:36.615 INFO:tasks.workunit.client.0.smithi142.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:25.509834+0000 0'0 26:21 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:50:23.423254+0000 0'0 2020-03-11T22:50:23.423254+0000 0 2020-03-11T22:50:36.616 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:50:36.616 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:36.616 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:50:36.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:50:36.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:50:36.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:50:36.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:50:36.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:50:36.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:50:36.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:50:36.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:50:36.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:50:36.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:50:36.627 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:50:36.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:36.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:50:36.628 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:50:36.959 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:36.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:50:36.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:36.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:50:36.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:50:36.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:50:36.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:50:36.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:50:36.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:50:36.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:50:36.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:36.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:50:36.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:50:37.298 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:37.299 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:50:37.299 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:37.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:50:37.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:50:37.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:50:37.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:50:37.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:50:37.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:50:37.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:50:37.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:50:37.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:50:37.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:50:37.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:50:37.304 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:50:37.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:37.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:50:37.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:37.304 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:50:37.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:37.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:50:37.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:50:37.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:50:37.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:50:37.789 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test3' does not exist 2020-03-11T22:50:37.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:50:37.797 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:50:37.798 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:50:37.799 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:50:37.799 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:50:37.800 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:50:37.800 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:50:38.605 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 size to 1 2020-03-11T22:50:38.613 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:50:39.185 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:50:39.193 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:50:39.828 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 2 size to 1 2020-03-11T22:50:39.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:50:40.579 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:50:40.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:50:40.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:40.588 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:40.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:40.589 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:40.589 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:40.590 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:40.590 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:40.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:40.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:40.591 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:40.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:40.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:40.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:40.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:40.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:41.047 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:41.047 INFO:tasks.workunit.client.0.smithi142.stderr:1' 2020-03-11T22:50:41.047 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:41.048 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:41.048 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:41.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:50:41.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:50:41.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:41.175 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:41.300 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:50:41.300 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2020-03-11T22:50:41.301 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:41.301 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:50:41.301 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:41.308 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:41.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:50:41.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:41.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:50:41.309 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:50:41.309 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:41.309 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:50:41.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836490 2020-03-11T22:50:41.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:42.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:42.649 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:43.010 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:50:43.010 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:43.011 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:50:43.011 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:43.012 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:43.013 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:50:43.013 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:43.014 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:50:43.014 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:50:43.014 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:50:43.015 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:43.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672968 2020-03-11T22:50:43.360 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:43.360 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:43.361 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:43.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:50:43.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:43.805 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:43.805 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:43.805 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:43.805 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:43.806 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:43.806 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:44.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:50:44.139 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:44.140 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:44.140 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:44.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:50:44.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:44.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:44.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:50:44.630 INFO:tasks.workunit.client.0.smithi142.stderr:10+0 records in 2020-03-11T22:50:44.630 INFO:tasks.workunit.client.0.smithi142.stderr:10+0 records out 2020-03-11T22:50:44.630 INFO:tasks.workunit.client.0.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0458071 s, 229 MB/s 2020-03-11T22:50:44.630 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:50:44.630 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:50:44.631 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:50:44.631 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:44.631 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:50:44.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:44.909 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.617 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.617 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.866 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.866 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.392 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.392 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.601 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.810 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.051 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.052 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.259 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.594 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.594 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.835 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.835 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.302 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.777 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.777 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.061 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.311 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.536 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.536 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.020 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.722 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.981 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.912 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.912 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.129 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.129 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.395 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:50:54.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:50:54.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:50:54.631 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.632 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:50:54.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:50:55.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:50:55.340 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.340 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:50:55.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:50:55.815 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.816 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:50:56.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:50:56.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:50:56.574 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.575 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:50:56.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:50:56.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:50:56.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:50:56.827 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:50:56.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:50:57.052 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.052 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:50:57.333 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.334 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:50:57.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:50:57.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.402 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.403 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.628 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.628 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.861 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.861 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.303 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.303 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.761 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.762 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.028 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.237 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.237 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.482 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.482 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.195 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.196 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.704 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.704 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.935 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.216 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.216 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.430 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.946 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.188 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.188 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.157 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.157 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.038 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.039 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.289 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.289 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.748 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.019 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.019 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.281 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.498 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.498 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.748 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.748 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.956 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.956 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.223 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.681 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:51:08.448 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.449 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:51:08.708 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:51:08.934 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:51:08.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:51:08.936 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:51:08.936 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:51:08.936 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:51:08.936 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:51:08.936 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:51:08.936 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:51:08.937 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:51:08.937 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:51:09.290 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:51:09.291 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:51:09.291 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:51:09.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:51:09.640 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:51:09.971 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:09.972 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:50:38.472310+0000 37'150 37:193 [1] 1 [1] 1 0'0 2020-03-11T22:50:21.034174+0000 0'0 2020-03-11T22:50:21.034174+0000 0 2020-03-11T22:51:09.972 INFO:tasks.workunit.client.0.smithi142.stdout:2.0 34 0 0 0 0 356515840 0 0 102 102 active+clean 2020-03-11T22:50:39.842056+0000 37'102 37:138 [0] 0 [0] 0 0'0 2020-03-11T22:50:23.423254+0000 0'0 2020-03-11T22:50:23.423254+0000 0 2020-03-11T22:51:09.972 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:51:09.972 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:09.973 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:51:09.981 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:51:09.982 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:51:10.611 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 size to 2 2020-03-11T22:51:10.619 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:51:11.866 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 2 size to 2 2020-03-11T22:51:11.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:51:16.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:51:16.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:16.878 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:16.878 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:16.878 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:16.878 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:16.878 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:51:16.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:51:16.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:16.978 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 2020-03-11T22:51:16.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:51:16.979 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:51:16.979 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:51:16.979 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:51:16.979 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 152, 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:51:16.980 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:51:16.981 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:51:16.981 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:51:16.981 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:51:16.981 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:51:16.981 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:51:16.981 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:51:16.981 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:51:16.982 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:51:16.982 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "2.0", 2020-03-11T22:51:16.982 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 153, 2020-03-11T22:51:16.982 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:51:16.982 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:51:16.982 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:51:16.982 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:51:16.983 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:51:16.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:51:16.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:51:16.984 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:51:16.984 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:16.984 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:16.984 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:16.985 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.0.asok 2020-03-11T22:51:16.985 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:51:16.985 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:51:17.077 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 2020-03-11T22:51:17.077 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:51:17.077 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:51:17.078 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:51:17.078 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:51:17.078 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:51:17.078 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:51:17.078 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:51:17.078 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "2.0", 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 153, 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:51:17.079 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:51:17.080 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:51:17.080 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:51:17.080 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:51:17.080 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:51:17.080 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:51:17.080 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:51:17.080 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 152, 2020-03-11T22:51:17.081 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:51:17.081 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:51:17.081 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:51:17.081 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:51:17.081 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:51:17.082 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:51:17.082 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:17.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:51:17.083 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:51:17.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:51:17.083 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:51:17.083 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:17.084 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:51:17.084 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:51:17.085 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:51:17.085 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:51:17.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:51:17.087 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:51:17.088 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:51:17.088 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:51:17.088 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:51:17.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:51:17.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:51:17.091 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:51:17.092 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:17.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:51:17.094 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:51:17.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:51:17.095 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:51:17.095 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:17.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:51:17.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:51:17.098 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:51:17.098 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:51:17.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:51:17.100 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:51:17.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:51:17.101 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:51:17.101 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:51:17.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:51:17.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:51:17.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:51:17.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:51:17.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:51:17.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:51:17.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:51:17.106 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:51:17.106 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:51:17.106 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:51:17.106 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:51:17.106 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:51:17.106 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:51:17.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:51:17.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:51:17.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:51:17.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:17.195 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:17.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:17.565 INFO:tasks.workunit.client.0.smithi142.stderr:1' 2020-03-11T22:51:17.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:17.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:17.566 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:17.692 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836499 2020-03-11T22:51:17.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499' 2020-03-11T22:51:17.693 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:17.693 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:17.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672978 2020-03-11T22:51:17.839 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2020-03-11T22:51:17.840 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:17.840 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836499 2020-03-11T22:51:17.840 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:17.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:17.842 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836499 2020-03-11T22:51:17.842 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:17.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836499 2020-03-11T22:51:17.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-03-11T22:51:17.843 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836499 2020-03-11T22:51:17.843 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:18.382 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836499 2020-03-11T22:51:18.382 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:18.383 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672978 2020-03-11T22:51:18.383 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:18.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:18.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672978 2020-03-11T22:51:18.385 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:18.386 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672978 2020-03-11T22:51:18.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672978 2020-03-11T22:51:18.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-03-11T22:51:18.386 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:18.774 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 42949672978 2020-03-11T22:51:18.775 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:51:18.775 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:18.775 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:19.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:51:19.261 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:19.261 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:19.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:19.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:19.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:19.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:19.262 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:19.597 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:19.598 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:19.598 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:19.598 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:20.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:20.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:51:20.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:20.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:51:20.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:20.050 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:20.150 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:20.151 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:20.152 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:20.152 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:20.152 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:20.152 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:20.152 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:20.152 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:20.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:20.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:20.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:20.559 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:21.006 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:21.006 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:21.006 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:21.006 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:21.007 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:21.007 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:21.007 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:21.007 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:21.008 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:21.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26207276 2020-03-11T22:51:21.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26207276 '!=' null 2020-03-11T22:51:21.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:21.461 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:21.461 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:21.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:21.561 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:21.562 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:21.562 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:21.562 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:21.562 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:21.563 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:21.563 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:21.901 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:21.901 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:21.902 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:21.902 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:22.376 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:22.376 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:22.377 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:22.377 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:22.377 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:22.377 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:22.377 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:22.378 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:22.378 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:22.948 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32499765 2020-03-11T22:51:22.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32499765 '!=' null 2020-03-11T22:51:22.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:22.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:22.949 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:23.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:23.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:23.050 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:23.050 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:23.051 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:23.051 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:23.051 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:23.051 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:23.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:23.446 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:23.447 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:23.447 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:23.899 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:23.899 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:23.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:23.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:23.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:23.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:23.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:23.901 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:23.901 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:24.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28037737 2020-03-11T22:51:24.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28037737 '!=' null 2020-03-11T22:51:24.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:24.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:24.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:24.471 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:24.472 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:24.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:24.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:24.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:24.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:24.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:24.473 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:24.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:24.844 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:24.845 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:24.845 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:25.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:25.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:25.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:25.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:25.990 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28037737 2020-03-11T22:51:25.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28037737 '!=' null 2020-03-11T22:51:25.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:25.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:25.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:26.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:26.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:26.092 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:26.093 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:26.093 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:26.093 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:26.093 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:26.093 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:26.430 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:26.431 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:26.431 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:26.431 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:26.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:26.884 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:26.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:26.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:26.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:26.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:26.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:26.886 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:26.886 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:27.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946292 2020-03-11T22:51:27.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946292 '!=' null 2020-03-11T22:51:27.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:27.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:27.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:27.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:27.584 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:27.584 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:27.585 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:27.585 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:27.585 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:27.585 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:27.950 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:27.951 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:27.951 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:27.951 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:28.462 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:28.462 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:28.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:28.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:28.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:28.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:28.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:28.464 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:28.464 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:28.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39314786 2020-03-11T22:51:28.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39314786 '!=' null 2020-03-11T22:51:28.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:28.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:28.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:29.071 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:29.071 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:29.072 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:29.072 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:29.073 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:29.073 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:29.073 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:29.073 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:29.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:29.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:29.414 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:29.415 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:29.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:29.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:29.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:29.973 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:29.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:29.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:29.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:29.974 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:29.975 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:30.549 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25336458 2020-03-11T22:51:30.549 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25336458 '!=' null 2020-03-11T22:51:30.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:30.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:30.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:30.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:30.650 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:30.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:30.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:30.651 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:30.652 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:30.652 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:30.652 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:31.049 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:31.049 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:31.050 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:31.050 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:31.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:31.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:31.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:31.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:31.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:31.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:31.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:31.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:31.656 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:32.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40189113 2020-03-11T22:51:32.319 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40189113 '!=' null 2020-03-11T22:51:32.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:32.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:32.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:32.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:32.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:32.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:32.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:32.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:32.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:32.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:32.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:32.765 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:32.766 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:32.766 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:32.767 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:33.218 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:33.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:33.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:33.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:33.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:33.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:33.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:33.220 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:33.220 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:34.025 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40189113 2020-03-11T22:51:34.025 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40189113 '!=' null 2020-03-11T22:51:34.025 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:34.026 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:34.026 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:34.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:34.126 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:34.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:34.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:34.127 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:34.128 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:34.128 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:34.128 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:34.473 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:34.474 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:34.474 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:34.474 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:34.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:34.926 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:34.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:34.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:34.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:34.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:34.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:34.928 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:34.928 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:35.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=30578658 2020-03-11T22:51:35.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 30578658 '!=' null 2020-03-11T22:51:35.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:35.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:35.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:35.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:35.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:35.609 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:35.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:35.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:35.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:35.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:35.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:35.947 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:35.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:35.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:35.948 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:36.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:36.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:36.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:36.419 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:36.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:36.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:36.420 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:36.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:36.421 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:36.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=38441657 2020-03-11T22:51:36.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 38441657 '!=' null 2020-03-11T22:51:36.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:36.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:36.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:36.979 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:36.979 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:36.980 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:36.980 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:36.980 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:36.981 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:36.981 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:36.981 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:37.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:37.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:37.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:37.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:37.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:37.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:37.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:37.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:37.855 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:37.855 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:38.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37568092 2020-03-11T22:51:38.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37568092 '!=' null 2020-03-11T22:51:38.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:38.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:38.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:38.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:38.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:38.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:38.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:38.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:38.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:38.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:38.514 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:38.872 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:38.873 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:38.873 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:38.873 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:39.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:39.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:39.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:39.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:39.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:39.441 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:39.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:39.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:39.442 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:39.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37568092 2020-03-11T22:51:39.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37568092 '!=' null 2020-03-11T22:51:39.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:39.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:39.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:40.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:40.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:40.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:40.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:40.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:40.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:40.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:40.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:40.406 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:40.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:40.853 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:40.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:40.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:40.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:40.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:40.854 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:40.855 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:40.855 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:41.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=35645578 2020-03-11T22:51:41.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 35645578 '!=' null 2020-03-11T22:51:41.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:41.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:41.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:41.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:41.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:41.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:41.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:41.422 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:41.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:41.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:41.423 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:41.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:51:41.753 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:41.754 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:41.754 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:42.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:51:42.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:51:42.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:42.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:42.330 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:42.330 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:42.330 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:42.330 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:42.331 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:42.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39314386 2020-03-11T22:51:42.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39314386 '!=' null 2020-03-11T22:51:42.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:42.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:42.775 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:42.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:42.876 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:42.876 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:42.876 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:42.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:42.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:42.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:42.877 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:43.209 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:43.210 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:43.211 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:43.211 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:43.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:51:43.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:51:43.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:51:43.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:51:43.658 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:43.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:43.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:43.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:43.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:43.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:43.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:43.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:43.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:43.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:44.090 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:44.091 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:44.091 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:44.091 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:44.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:51:44.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:44.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:44.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:44.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:44.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:44.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:44.535 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:44.535 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:44.982 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=38315542 2020-03-11T22:51:44.983 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 38315542 '!=' null 2020-03-11T22:51:44.983 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:44.983 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:44.983 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:45.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:45.083 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:45.084 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:45.084 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:45.085 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:45.085 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:45.085 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:45.085 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:45.416 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:51:45.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:45.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:45.417 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:45.858 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:51:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:51:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:51:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:51:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:51:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:51:45.859 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:51:45.860 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:51:45.860 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:51:46.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704999 2020-03-11T22:51:46.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704999 '!=' null 2020-03-11T22:51:46.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:51:46.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:51:46.305 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:51:46.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:51:46.406 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:51:46.406 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:51:46.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:51:46.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:51:46.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:51:46.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:51:46.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:51:46.744 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:51:46.745 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:51:46.745 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:51:46.745 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:51:47.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:51:47.190 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:51:47.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:51:47.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:51:47.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:51:47.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:51:47.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:51:47.191 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:51:47.191 INFO:tasks.workunit.client.0.smithi142.stdout:TEST PASSED 2020-03-11T22:51:47.672 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test1' does not exist 2020-03-11T22:51:47.680 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:51:47.680 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:51:47.681 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:51:48.283 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test2' does not exist 2020-03-11T22:51:48.291 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:51:48.292 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:48.292 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:48.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:48.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:48.292 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:48.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:48.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:51:48.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:51:48.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:51:48.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:51:48.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:51:48.400 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:48.400 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:48.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:48.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:48.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:48.403 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:48.403 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:51:48.404 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:51:48.404 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:51:48.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:51:48.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:51:48.405 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:51:48.407 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:48.407 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:51:48.407 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:51:48.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:48.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:51:48.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:51:48.413 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:51:48.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:51:48.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:48.420 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:48.421 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55987 2020-03-11T22:51:48.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:51:48.422 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:51:48.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:51:48.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:51:48.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:51:48.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:51:48.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:51:48.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:51:48.423 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:51:48.424 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:51:48.424 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:51:48.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:51:48.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:51:48.424 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:51:48.425 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:51:48.425 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:51:48.426 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:51:48.426 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:51:48.426 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:51:48.426 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:51:48.426 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:51:48.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:48.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:51:48.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:51:48.427 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:51:48.428 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:51:48.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:51:48.429 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:51:48.430 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:51:48.430 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:48.430 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:48.430 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55987 2020-03-11T22:51:48.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:51:48.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:51:48.431 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:51:48.432 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:51:48.432 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:48.432 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:48.432 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.55987 2020-03-11T22:51:48.433 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:51:48.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:51:48.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:51:48.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:51:48.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:51:48.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:51:48.434 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:51:48.435 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:51:48.435 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:51:48.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:51:48.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:51:48.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:51:48.436 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:51:48.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:51:48.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:51:48.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:51:48.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:51:48.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:51:48.729 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:51:48.729 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:48.729 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:48.729 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:48.729 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:48.730 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:48.730 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:51:48.730 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:51:48.879 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:51:48.881 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:51:48.881 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:51:48.881 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:51:48.882 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:51:48.882 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:51:48.882 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:51:48.882 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:51:48.882 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:51:48.882 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:48.882 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:48.883 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:48.883 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-mon.a.asok 2020-03-11T22:51:48.883 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:51:48.883 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55987/ceph-mon.a.asok config get fsid 2020-03-11T22:51:48.968 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:51:48.968 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:51:48.968 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:51:48.969 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:51:48.969 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:51:48.969 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:51:48.969 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:51:48.969 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:51:48.969 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:48.969 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:48.970 INFO:tasks.workunit.client.0.smithi142.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:48.970 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-mon.a.asok 2020-03-11T22:51:48.970 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:51:48.970 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.55987/ceph-mon.a.asok config get mon_host 2020-03-11T22:51:49.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:51:49.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:51:49.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:51:49.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:51:49.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:51:49.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:51:49.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:51:49.929 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:51:49.929 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:49.929 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:49.929 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:49.930 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:49.930 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:49.930 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:51:49.931 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:51:49.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:51:49.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:51:49.953 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:51:49.954 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:51:49.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:49.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:51:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:51:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:51:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:51:49.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:51:49.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:51:49.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:49.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:49.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:49.961 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:49.961 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:49.961 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:49.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:49.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:49.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:49.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:49.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:49.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:51:49.963 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:49.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=83b87654-1dc1-4f18-979c-01feb8596e32 2020-03-11T22:51:49.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 83b87654-1dc1-4f18-979c-01feb8596e32' 2020-03-11T22:51:49.964 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:49.964 INFO:tasks.workunit.client.0.smithi142.stdout:add osd0 83b87654-1dc1-4f18-979c-01feb8596e32 2020-03-11T22:51:49.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCFa2leRAvrORAA5y371CWiXMKKMNjjEx4W5g== 2020-03-11T22:51:49.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCFa2leRAvrORAA5y371CWiXMKKMNjjEx4W5g=="}' 2020-03-11T22:51:49.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 83b87654-1dc1-4f18-979c-01feb8596e32 -i td/osd-backfill-prio/0/new.json 2020-03-11T22:51:50.311 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:51:50.320 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:51:50.321 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCFa2leRAvrORAA5y371CWiXMKKMNjjEx4W5g== --osd-uuid 83b87654-1dc1-4f18-979c-01feb8596e32 2020-03-11T22:51:50.347 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:50.344+0000 7f9fe77e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:50.349 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:50.346+0000 7f9fe77e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:50.351 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:50.348+0000 7f9fe77e6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:50.351 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:50.348+0000 7f9fe77e6ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:51:50.352 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:50.349+0000 7f9fe77e6ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:51:50.352 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:50.349+0000 7f9fe77e6ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:51:50.387 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:50.384+0000 7f9fe77e6ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:51:53.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:51:53.385 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:53.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:51:53.386 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:53.386 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd0 key to auth repository 2020-03-11T22:51:53.833 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:51:53.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:53.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:53.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:53.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:53.834 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.0 2020-03-11T22:51:53.857 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:53.854+0000 7f59018daec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:53.859 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:53.856+0000 7f59018daec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:53.860 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:53.857+0000 7f59018daec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:54.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:51:54.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:51:54.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:51:54.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:51:54.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:51:54.183 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:54.184 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:51:54.184 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:54.184 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:51:54.184 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:51:54.376 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:54.373+0000 7f59018daec0 -1 Falling back to public interface 2020-03-11T22:51:54.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:51:55.127 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:55.124+0000 7f59018daec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:51:55.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:51:55.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:51:55.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:51:55.536 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:51:55.536 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:51:55.536 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:51:55.885 INFO:tasks.workunit.client.0.smithi142.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/62856,v1:127.0.0.1:6803/62856] [v2:127.0.0.1:6804/62856,v1:127.0.0.1:6805/62856] exists,up 83b87654-1dc1-4f18-979c-01feb8596e32 2020-03-11T22:51:55.885 INFO:tasks.workunit.client.0.smithi142.stdout:add osd1 4afac70f-9edb-490d-8992-111acf2c140b 2020-03-11T22:51:55.885 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:51:55.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:51:55.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:51:55.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:51:55.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:51:55.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:51:55.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:51:55.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:51:55.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:51:55.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:51:55.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:51:55.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:51:55.888 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:51:55.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:51:55.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:51:55.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:51:55.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:51:55.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:51:55.889 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:51:55.889 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:51:55.890 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:51:55.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:51:55.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:51:55.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:51:55.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:51:55.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:51:55.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:51:55.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:55.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:51:55.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:51:55.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:51:55.892 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:51:55.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4afac70f-9edb-490d-8992-111acf2c140b 2020-03-11T22:51:55.892 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 4afac70f-9edb-490d-8992-111acf2c140b' 2020-03-11T22:51:55.892 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:51:55.900 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCLa2leksCFNRAAKKdwukvMa+9EFVwnIGF6CA== 2020-03-11T22:51:55.901 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCLa2leksCFNRAAKKdwukvMa+9EFVwnIGF6CA=="}' 2020-03-11T22:51:55.901 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4afac70f-9edb-490d-8992-111acf2c140b -i td/osd-backfill-prio/1/new.json 2020-03-11T22:51:56.400 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:51:56.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:51:56.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCLa2leksCFNRAAKKdwukvMa+9EFVwnIGF6CA== --osd-uuid 4afac70f-9edb-490d-8992-111acf2c140b 2020-03-11T22:51:56.437 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:56.434+0000 7f5639440ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:56.438 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:56.435+0000 7f5639440ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:56.439 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:56.436+0000 7f5639440ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:56.440 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:56.437+0000 7f5639440ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:51:56.440 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:56.437+0000 7f5639440ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:51:56.440 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:56.437+0000 7f5639440ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:51:56.516 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:56.514+0000 7f5639440ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:51:59.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:51:59.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:51:59.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:51:59.448 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:51:59.448 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd1 key to auth repository 2020-03-11T22:51:59.909 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:51:59.909 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:51:59.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:51:59.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:51:59.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:51:59.910 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.1 2020-03-11T22:51:59.932 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:59.929+0000 7f5efe730ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:59.934 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:59.931+0000 7f5efe730ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:51:59.935 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:51:59.933+0000 7f5efe730ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:00.259 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:52:00.259 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:00.259 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:52:00.259 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:00.260 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:00.260 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:52:00.450 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:00.447+0000 7f5efe730ec0 -1 Falling back to public interface 2020-03-11T22:52:00.602 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:01.189 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:01.186+0000 7f5efe730ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:52:01.603 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:01.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:01.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:01.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:01.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:52:01.604 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:52:01.958 INFO:tasks.workunit.client.0.smithi142.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63211,v1:127.0.0.1:6811/63211] [v2:127.0.0.1:6812/63211,v1:127.0.0.1:6813/63211] exists,up 4afac70f-9edb-490d-8992-111acf2c140b 2020-03-11T22:52:01.958 INFO:tasks.workunit.client.0.smithi142.stdout:add osd2 6200a47e-403b-46f4-8a94-18016da009cf 2020-03-11T22:52:01.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:01.958 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:01.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:01.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:01.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:52:01.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:52:01.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:01.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:52:01.959 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:01.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:52:01.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:52:01.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:01.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:01.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:01.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:52:01.960 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:52:01.961 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:01.961 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:52:01.961 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:52:01.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:01.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:01.961 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:01.961 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:01.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:52:01.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:52:01.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:01.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:01.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:01.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:01.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:01.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:52:01.964 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:01.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6200a47e-403b-46f4-8a94-18016da009cf 2020-03-11T22:52:01.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 6200a47e-403b-46f4-8a94-18016da009cf' 2020-03-11T22:52:01.964 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:01.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCRa2leErrvORAAh3yXy+WNii9AozruA+nEkQ== 2020-03-11T22:52:01.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCRa2leErrvORAAh3yXy+WNii9AozruA+nEkQ=="}' 2020-03-11T22:52:01.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6200a47e-403b-46f4-8a94-18016da009cf -i td/osd-backfill-prio/2/new.json 2020-03-11T22:52:02.486 INFO:tasks.workunit.client.0.smithi142.stdout:2 2020-03-11T22:52:02.496 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:52:02.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCRa2leErrvORAAh3yXy+WNii9AozruA+nEkQ== --osd-uuid 6200a47e-403b-46f4-8a94-18016da009cf 2020-03-11T22:52:02.522 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:02.519+0000 7f1c7782fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:02.523 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:02.521+0000 7f1c7782fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:02.526 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:02.523+0000 7f1c7782fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:02.526 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:02.523+0000 7f1c7782fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:52:02.526 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:02.523+0000 7f1c7782fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:52:02.527 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:02.523+0000 7f1c7782fec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:52:02.609 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:02.606+0000 7f1c7782fec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:52:05.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:52:05.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:05.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:52:05.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:05.649 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd2 key to auth repository 2020-03-11T22:52:06.111 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.2 2020-03-11T22:52:06.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:52:06.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:06.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:06.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:06.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:06.135 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:06.132+0000 7fda8b532ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:06.137 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:06.134+0000 7fda8b532ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:06.137 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:06.135+0000 7fda8b532ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:06.462 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:52:06.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:06.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:52:06.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:06.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:06.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:06.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:06.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:06.464 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:06.464 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:52:06.653 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:06.650+0000 7fda8b532ec0 -1 Falling back to public interface 2020-03-11T22:52:06.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:07.436 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:07.433+0000 7fda8b532ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:52:07.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:07.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:07.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:07.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:07.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:52:07.809 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:52:08.169 INFO:tasks.workunit.client.0.smithi142.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63566,v1:127.0.0.1:6819/63566] [v2:127.0.0.1:6820/63566,v1:127.0.0.1:6821/63566] exists,up 6200a47e-403b-46f4-8a94-18016da009cf 2020-03-11T22:52:08.169 INFO:tasks.workunit.client.0.smithi142.stdout:add osd3 bafb2d50-505f-446d-92e6-8add8cb0be0d 2020-03-11T22:52:08.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:08.169 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:08.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:08.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:08.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:52:08.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:52:08.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:08.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:52:08.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:08.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:52:08.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:52:08.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:08.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:08.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:08.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:52:08.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:52:08.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:08.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:52:08.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:52:08.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:08.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:08.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:08.173 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:08.173 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:08.173 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:52:08.173 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:52:08.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:52:08.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:08.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:08.174 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:08.175 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:52:08.175 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:08.175 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=bafb2d50-505f-446d-92e6-8add8cb0be0d 2020-03-11T22:52:08.175 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 bafb2d50-505f-446d-92e6-8add8cb0be0d' 2020-03-11T22:52:08.175 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:08.185 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCYa2leuSbcChAAikDaZTj1Xs+f7hD2aJwe2w== 2020-03-11T22:52:08.185 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCYa2leuSbcChAAikDaZTj1Xs+f7hD2aJwe2w=="}' 2020-03-11T22:52:08.185 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new bafb2d50-505f-446d-92e6-8add8cb0be0d -i td/osd-backfill-prio/3/new.json 2020-03-11T22:52:08.635 INFO:tasks.workunit.client.0.smithi142.stdout:3 2020-03-11T22:52:08.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:52:08.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCYa2leuSbcChAAikDaZTj1Xs+f7hD2aJwe2w== --osd-uuid bafb2d50-505f-446d-92e6-8add8cb0be0d 2020-03-11T22:52:08.670 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:08.667+0000 7f48aaca2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:08.672 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:08.669+0000 7f48aaca2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:08.673 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:08.671+0000 7f48aaca2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:08.674 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:08.671+0000 7f48aaca2ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:52:08.674 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:08.671+0000 7f48aaca2ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:52:08.674 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:08.671+0000 7f48aaca2ec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:52:08.717 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:08.715+0000 7f48aaca2ec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:52:11.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:52:11.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:11.678 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd3 key to auth repository 2020-03-11T22:52:11.678 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:52:11.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:12.138 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.3 2020-03-11T22:52:12.138 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:52:12.138 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:12.138 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:12.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:12.139 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:12.161 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:12.157+0000 7fdfc06f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:12.163 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:12.159+0000 7fdfc06f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:12.164 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:12.160+0000 7fdfc06f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:12.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:52:12.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:12.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:52:12.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:12.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:12.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:12.485 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:12.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:12.486 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:12.486 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:52:12.679 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:12.675+0000 7fdfc06f9ec0 -1 Falling back to public interface 2020-03-11T22:52:12.842 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:13.609 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:13.605+0000 7fdfc06f9ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:52:13.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:13.843 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:13.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:13.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:13.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:13.844 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:52:14.202 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:15.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:15.203 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:15.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:52:15.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:15.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:52:15.204 INFO:tasks.workunit.client.0.smithi142.stdout:2 2020-03-11T22:52:15.564 INFO:tasks.workunit.client.0.smithi142.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/63921,v1:127.0.0.1:6827/63921] [v2:127.0.0.1:6828/63921,v1:127.0.0.1:6829/63921] exists,up bafb2d50-505f-446d-92e6-8add8cb0be0d 2020-03-11T22:52:15.564 INFO:tasks.workunit.client.0.smithi142.stdout:add osd4 e35b9010-8e5c-4ee0-85bb-542f0621158e 2020-03-11T22:52:15.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:52:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:52:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:52:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:52:15.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:52:15.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:52:15.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:52:15.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:52:15.566 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:52:15.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:52:15.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:52:15.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:52:15.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:52:15.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:52:15.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:52:15.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:52:15.568 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:52:15.568 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:52:15.568 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:52:15.568 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:52:15.568 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:15.568 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:52:15.568 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:52:15.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' 2020-03-11T22:52:15.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:52:15.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:52:15.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:52:15.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:52:15.569 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:52:15.570 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:52:15.570 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:52:15.570 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:15.570 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:52:15.570 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:52:15.570 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:52:15.570 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:52:15.571 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e35b9010-8e5c-4ee0-85bb-542f0621158e 2020-03-11T22:52:15.571 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 e35b9010-8e5c-4ee0-85bb-542f0621158e' 2020-03-11T22:52:15.571 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:52:15.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCfa2lelCtpIhAAicJSDbT4/wJl4xirgG0zLQ== 2020-03-11T22:52:15.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCfa2lelCtpIhAAicJSDbT4/wJl4xirgG0zLQ=="}' 2020-03-11T22:52:15.580 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e35b9010-8e5c-4ee0-85bb-542f0621158e -i td/osd-backfill-prio/4/new.json 2020-03-11T22:52:16.046 INFO:tasks.workunit.client.0.smithi142.stdout:4 2020-03-11T22:52:16.056 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:52:16.057 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCfa2lelCtpIhAAicJSDbT4/wJl4xirgG0zLQ== --osd-uuid e35b9010-8e5c-4ee0-85bb-542f0621158e 2020-03-11T22:52:16.082 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:16.078+0000 7fe464309ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:16.084 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:16.080+0000 7fe464309ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:16.086 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:16.082+0000 7fe464309ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:16.086 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:16.082+0000 7fe464309ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:52:16.086 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:16.082+0000 7fe464309ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:52:16.087 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:16.082+0000 7fe464309ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:52:16.125 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:16.122+0000 7fe464309ec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:52:19.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:52:19.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:52:19.096 INFO:tasks.workunit.client.0.smithi142.stdout:adding osd4 key to auth repository 2020-03-11T22:52:19.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:52:19.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:52:19.559 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:52:19.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:52:19.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=7756c544-fc18-4cc0-a699-9a2246573cff --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.55987/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:52:19.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:52:19.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:52:19.560 INFO:tasks.workunit.client.0.smithi142.stdout:start osd.4 2020-03-11T22:52:19.583 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:19.579+0000 7ff8d4283ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:19.585 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:19.581+0000 7ff8d4283ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:19.587 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:19.583+0000 7ff8d4283ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:52:19.907 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:52:19.907 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:52:19.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:52:19.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:52:19.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:52:19.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:19.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:52:19.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:19.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:52:19.909 INFO:tasks.workunit.client.0.smithi142.stdout:0 2020-03-11T22:52:20.101 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:20.097+0000 7ff8d4283ec0 -1 Falling back to public interface 2020-03-11T22:52:20.255 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:52:20.836 INFO:tasks.workunit.client.0.smithi142.stderr:2020-03-11T22:52:20.832+0000 7ff8d4283ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:52:21.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:52:21.257 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:52:21.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:52:21.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:52:21.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:52:21.258 INFO:tasks.workunit.client.0.smithi142.stdout:1 2020-03-11T22:52:21.616 INFO:tasks.workunit.client.0.smithi142.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64312,v1:127.0.0.1:6835/64312] [v2:127.0.0.1:6836/64312,v1:127.0.0.1:6837/64312] exists,up e35b9010-8e5c-4ee0-85bb-542f0621158e 2020-03-11T22:52:21.616 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:52:21.616 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:52:21.616 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:52:21.617 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:52:21.617 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:21.617 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:52:21.618 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:52:22.059 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test1' already exists 2020-03-11T22:52:22.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:23.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:52:23.707 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 1 size to 2 2020-03-11T22:52:23.715 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:23.716 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:52:23.716 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:52:24.417 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test2' already exists 2020-03-11T22:52:24.425 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:25.427 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:52:26.390 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 2 size to 2 2020-03-11T22:52:26.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:26.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:52:26.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:52:26.905 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test3' already exists 2020-03-11T22:52:26.913 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:27.914 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:52:28.729 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 3 size to 2 2020-03-11T22:52:28.737 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:28.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:52:28.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:52:29.205 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test4' already exists 2020-03-11T22:52:29.212 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:30.214 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:52:30.818 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 4 size to 2 2020-03-11T22:52:30.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:30.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:52:30.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:52:31.523 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test5' already exists 2020-03-11T22:52:31.530 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:32.532 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:52:33.366 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 5 size to 2 2020-03-11T22:52:33.374 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:33.374 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:52:33.374 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:52:33.922 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test6' already exists 2020-03-11T22:52:33.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:34.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:52:35.504 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 6 size to 2 2020-03-11T22:52:35.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:35.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:52:35.513 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:52:36.096 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test7' already exists 2020-03-11T22:52:36.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:37.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:52:37.852 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 7 size to 2 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:52:37.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:52:38.334 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test8' already exists 2020-03-11T22:52:38.342 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:39.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:52:39.991 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 8 size to 2 2020-03-11T22:52:39.999 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:39.999 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:52:39.999 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:52:40.649 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test9' already exists 2020-03-11T22:52:40.657 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:41.659 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:52:42.315 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 9 size to 2 2020-03-11T22:52:42.323 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:42.323 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:52:42.324 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:52:42.924 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test10' already exists 2020-03-11T22:52:42.932 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:52:43.934 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:52:44.554 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 10 size to 2 2020-03-11T22:52:44.562 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:52:49.563 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:52:49.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:52:49.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:52:49.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:52:49.564 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:52:49.564 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:52:49.565 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:52:49.565 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:52:49.566 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:52:49.566 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:52:49.566 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:52:49.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:52:49.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:52:49.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:52:49.667 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:52:49.667 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:52:50.016 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:52:50.017 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:52:50.017 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:52:50.017 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:52:50.017 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:52:50.017 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:52:50.018 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:50.018 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:52:50.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:52:50.142 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:52:50.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:50.143 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:52:50.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672972 2020-03-11T22:52:50.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2020-03-11T22:52:50.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:50.269 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:52:50.395 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509452 2020-03-11T22:52:50.395 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509452' 2020-03-11T22:52:50.395 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:50.395 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:52:50.520 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345930 2020-03-11T22:52:50.520 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509452 3-85899345930' 2020-03-11T22:52:50.520 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:50.520 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:52:50.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182408 2020-03-11T22:52:50.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509452 3-85899345930 4-107374182408' 2020-03-11T22:52:50.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:50.646 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:52:50.646 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:50.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:52:50.646 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:52:50.647 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:50.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:52:50.648 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:52:50.648 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:50.648 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:52:50.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836494 2020-03-11T22:52:50.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:52:51.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:52:51.992 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:52.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-11T22:52:52.348 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:52.349 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672972 2020-03-11T22:52:52.349 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:52.350 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:52.350 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672972 2020-03-11T22:52:52.350 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:52.351 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672972 2020-03-11T22:52:52.351 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2020-03-11T22:52:52.351 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:52.351 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672972 2020-03-11T22:52:52.699 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672972 -lt 42949672972 2020-03-11T22:52:52.699 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:52.699 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509452 2020-03-11T22:52:52.699 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:52.700 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:52.700 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509452 2020-03-11T22:52:52.701 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:52.701 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509452 2020-03-11T22:52:52.702 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-03-11T22:52:52.702 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:52.702 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509452 2020-03-11T22:52:53.070 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509452 -lt 64424509452 2020-03-11T22:52:53.071 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:53.071 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345930 2020-03-11T22:52:53.071 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:53.072 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:53.072 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345930 2020-03-11T22:52:53.072 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:53.073 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345930 2020-03-11T22:52:53.074 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2020-03-11T22:52:53.075 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:53.075 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345930 2020-03-11T22:52:53.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345930 -lt 85899345930 2020-03-11T22:52:53.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:53.447 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182408 2020-03-11T22:52:53.447 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:53.448 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:53.448 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182408 2020-03-11T22:52:53.448 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:53.449 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182408 2020-03-11T22:52:53.449 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2020-03-11T22:52:53.449 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:53.450 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182408 2020-03-11T22:52:53.828 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182409 -lt 107374182408 2020-03-11T22:52:53.828 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:52:53.828 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:53.828 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:54.272 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:52:54.272 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:54.272 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:54.272 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:54.273 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:54.273 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:54.273 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:54.273 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:54.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:52:54.609 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:54.609 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:54.609 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:55.057 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:52:55.057 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:52:55.058 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:52:55.058 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:52:55.388 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:52:55.389 INFO:tasks.workunit.client.0.smithi142.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:44.966756+0000 0'0 78:14 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:52:42.638126+0000 0'0 2020-03-11T22:52:42.638126+0000 0 2020-03-11T22:52:55.389 INFO:tasks.workunit.client.0.smithi142.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:42.435901+0000 0'0 78:24 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:52:40.395085+0000 0'0 2020-03-11T22:52:40.395085+0000 0 2020-03-11T22:52:55.389 INFO:tasks.workunit.client.0.smithi142.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:40.656067+0000 0'0 78:28 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:52:38.177907+0000 0'0 2020-03-11T22:52:38.177907+0000 0 2020-03-11T22:52:55.389 INFO:tasks.workunit.client.0.smithi142.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:38.454940+0000 0'0 78:30 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:52:35.841059+0000 0'0 2020-03-11T22:52:35.841059+0000 0 2020-03-11T22:52:55.389 INFO:tasks.workunit.client.0.smithi142.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:26.184790+0000 0'0 78:52 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:52:24.128403+0000 0'0 2020-03-11T22:52:24.128403+0000 0 2020-03-11T22:52:55.389 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:24.187916+0000 0'0 78:54 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:52:21.881202+0000 0'0 2020-03-11T22:52:21.881202+0000 0 2020-03-11T22:52:55.389 INFO:tasks.workunit.client.0.smithi142.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:29.521237+0000 0'0 78:45 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:52:26.720637+0000 0'0 2020-03-11T22:52:26.720637+0000 0 2020-03-11T22:52:55.390 INFO:tasks.workunit.client.0.smithi142.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:31.345080+0000 0'0 78:43 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:52:29.032924+0000 0'0 2020-03-11T22:52:29.032924+0000 0 2020-03-11T22:52:55.390 INFO:tasks.workunit.client.0.smithi142.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:33.273091+0000 0'0 78:40 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:52:31.172057+0000 0'0 2020-03-11T22:52:31.172057+0000 0 2020-03-11T22:52:55.390 INFO:tasks.workunit.client.0.smithi142.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:52:36.432015+0000 0'0 78:34 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:52:33.711397+0000 0'0 2020-03-11T22:52:33.711397+0000 0 2020-03-11T22:52:55.390 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:52:55.390 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:52:55.391 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:52:55.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:52:55.399 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:52:55.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:52:55.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:52:55.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:52:55.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:52:55.400 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:52:55.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:52:55.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:52:55.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:52:55.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:52:55.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:52:55.401 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:52:55.401 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:55.402 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:52:55.402 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:55.736 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:55.736 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:52:55.736 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:55.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:52:55.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:52:55.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:52:55.739 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:52:55.739 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:52:55.739 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:52:55.739 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:52:55.739 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:55.739 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:52:55.739 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:56.101 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:56.102 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:52:56.102 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:56.103 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:52:56.103 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:56.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:52:56.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:52:56.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:56.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:52:56.104 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:56.441 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:56.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:52:56.444 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:56.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:52:56.444 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:56.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:52:56.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:52:56.445 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:52:56.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:52:56.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:52:56.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:56.446 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:52:56.447 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:56.778 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:56.779 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:52:56.779 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:56.780 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:52:56.781 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:56.781 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:52:56.781 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:52:56.781 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:56.782 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:52:56.782 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:57.121 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:57.122 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:52:57.122 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:57.123 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:52:57.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:57.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:52:57.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:52:57.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:57.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:52:57.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:57.463 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:57.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:52:57.464 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:57.465 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:52:57.465 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:57.465 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:52:57.465 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:52:57.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:57.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:52:57.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:52:57.802 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:52:57.803 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:52:57.803 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:52:57.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:52:57.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:52:57.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:52:57.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:52:57.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:52:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:52:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:52:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:52:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:52:57.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:52:57.806 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:52:57.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:57.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:52:57.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:57.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:52:57.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:52:57.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:52:57.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:52:58.261 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test2' does not exist 2020-03-11T22:52:58.269 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:58.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:52:58.270 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:58.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:52:58.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:52:58.271 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:52:58.272 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:52:58.729 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test4' does not exist 2020-03-11T22:52:58.737 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:58.737 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:52:58.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:52:58.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:52:58.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:52:59.220 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test5' does not exist 2020-03-11T22:52:59.228 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:59.228 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:52:59.228 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:52:59.229 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:52:59.229 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:52:59.825 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test6' does not exist 2020-03-11T22:52:59.833 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:59.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:52:59.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:52:59.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:52:59.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:52:59.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:52:59.834 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:53:00.318 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test8' does not exist 2020-03-11T22:53:00.326 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:53:00.326 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:53:00.327 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:53:00.327 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:53:00.327 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:53:00.800 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test9' does not exist 2020-03-11T22:53:00.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:53:00.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:53:00.808 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:53:00.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:53:00.809 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:53:01.364 INFO:tasks.workunit.client.0.smithi142.stderr:pool 'test10' does not exist 2020-03-11T22:53:01.372 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:53:02.259 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 3 size to 1 2020-03-11T22:53:02.267 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:53:02.903 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 7 size to 1 2020-03-11T22:53:02.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:53:02.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:53:02.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:53:02.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:53:02.915 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:53:02.916 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:53:02.916 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:53:02.916 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:53:02.916 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:53:02.917 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:53:02.917 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:53:03.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:53:03.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:53:03.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:53:03.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:53:03.004 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:53:03.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:53:03.369 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:53:03.370 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:53:03.370 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:53:03.371 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:53:03.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:53:03.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:03.371 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:53:03.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836499 2020-03-11T22:53:03.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499' 2020-03-11T22:53:03.497 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:03.497 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:53:03.623 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672977 2020-03-11T22:53:03.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2020-03-11T22:53:03.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:03.624 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:53:03.748 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509456 2020-03-11T22:53:03.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509456' 2020-03-11T22:53:03.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:03.749 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:53:03.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345934 2020-03-11T22:53:03.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509456 3-85899345934' 2020-03-11T22:53:03.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:53:03.875 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:53:04.000 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182413 2020-03-11T22:53:04.001 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509456 3-85899345934 4-107374182413' 2020-03-11T22:53:04.001 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:04.002 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836499 2020-03-11T22:53:04.002 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:04.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:53:04.003 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836499 2020-03-11T22:53:04.003 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:04.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836499 2020-03-11T22:53:04.004 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-03-11T22:53:04.004 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:04.004 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836499 2020-03-11T22:53:04.349 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836499 2020-03-11T22:53:04.350 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:53:05.350 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:53:05.351 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:53:05.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836499 2020-03-11T22:53:05.710 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:05.710 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672977 2020-03-11T22:53:05.710 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:05.711 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:53:05.711 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672977 2020-03-11T22:53:05.711 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:05.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672977 2020-03-11T22:53:05.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2020-03-11T22:53:05.712 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:53:05.713 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672977 2020-03-11T22:53:06.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672977 -lt 42949672977 2020-03-11T22:53:06.054 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509456 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509456 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:06.057 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509456 2020-03-11T22:53:06.057 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2020-03-11T22:53:06.057 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:53:06.057 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509456 2020-03-11T22:53:06.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509457 -lt 64424509456 2020-03-11T22:53:06.406 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:06.406 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345934 2020-03-11T22:53:06.406 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:06.407 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:53:06.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345934 2020-03-11T22:53:06.407 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:06.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345934 2020-03-11T22:53:06.408 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2020-03-11T22:53:06.408 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:53:06.409 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345934 2020-03-11T22:53:06.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345935 -lt 85899345934 2020-03-11T22:53:06.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:53:06.751 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182413 2020-03-11T22:53:06.751 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:53:06.752 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:53:06.752 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182413 2020-03-11T22:53:06.752 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:53:06.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182413 2020-03-11T22:53:06.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2020-03-11T22:53:06.753 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:53:06.753 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182413 2020-03-11T22:53:07.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182413 -lt 107374182413 2020-03-11T22:53:07.100 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:53:07.100 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:07.100 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:07.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:53:07.544 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:07.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:07.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:07.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:07.545 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:07.545 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:07.545 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:07.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:53:07.879 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:07.879 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:07.879 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:08.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:53:08.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:53:08.323 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:53:08.323 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:53:08.369 INFO:tasks.workunit.client.0.smithi142.stderr:10+0 records in 2020-03-11T22:53:08.369 INFO:tasks.workunit.client.0.smithi142.stderr:10+0 records out 2020-03-11T22:53:08.370 INFO:tasks.workunit.client.0.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0460926 s, 227 MB/s 2020-03-11T22:53:08.370 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:53:08.370 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:53:08.370 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:53:08.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:08.371 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:53:08.763 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:08.763 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:53:09.146 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:09.147 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:53:09.605 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:09.605 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:53:10.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:10.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:53:10.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:10.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:53:10.930 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:10.931 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:53:11.389 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:11.389 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:53:11.832 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:11.832 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:53:12.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:12.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:53:12.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:12.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:53:13.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:13.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:53:13.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:13.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:53:14.298 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:14.298 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:53:14.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:14.690 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:53:15.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:15.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:53:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:15.565 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:53:15.994 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:15.994 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:53:16.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:16.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:53:16.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:16.805 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:53:17.240 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:17.240 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:53:17.682 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:17.683 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:53:18.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:18.101 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:53:18.490 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:18.491 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:53:18.890 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:18.891 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:53:19.333 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:19.333 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:53:19.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:19.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:53:20.207 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:20.208 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:53:20.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:20.649 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:53:21.057 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:21.058 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:53:21.511 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:21.511 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:53:21.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:21.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:53:22.330 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:22.330 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:53:22.773 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:22.773 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:53:23.141 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:23.141 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:53:23.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:23.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:53:24.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:24.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:53:24.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:24.503 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:53:24.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:24.929 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:53:25.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:25.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:53:25.866 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:25.867 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:53:26.350 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:26.350 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:53:26.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:26.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:53:27.167 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:27.167 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:53:27.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:27.567 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:53:27.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:27.975 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:53:28.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:28.409 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:53:28.825 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:28.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:53:29.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:29.251 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:53:29.694 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:53:29.695 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:53:29.695 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:53:29.695 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:53:29.696 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:29.696 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:53:29.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:29.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:53:30.253 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:30.253 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:53:30.475 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:30.475 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:53:30.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:30.675 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:53:30.933 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:30.933 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:53:31.150 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:31.150 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:53:31.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:31.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:53:31.700 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:31.700 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:53:31.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:31.967 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:53:32.184 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:32.184 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:53:32.450 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:32.450 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:53:32.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:32.709 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:53:32.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:32.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:53:33.176 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:33.176 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:53:33.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:33.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:53:33.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:33.734 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:53:33.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:33.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:53:34.209 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:34.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:53:34.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:34.463 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:53:34.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:34.727 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:53:34.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:34.968 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:53:35.240 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:35.240 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:53:35.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:35.466 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:53:35.700 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:35.700 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:53:35.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:35.940 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:53:36.185 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:36.185 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:53:36.390 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:36.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:53:36.590 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:36.591 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:53:36.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:36.849 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:53:37.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:37.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:53:37.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:37.391 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:53:37.616 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:37.616 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:53:37.882 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:37.883 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:53:38.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:38.133 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:53:38.398 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:38.398 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:53:38.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:38.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:53:38.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:38.875 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:53:39.091 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:39.092 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:53:39.459 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:39.459 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:53:39.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:39.667 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:53:39.910 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:39.911 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:53:40.125 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:40.125 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:53:40.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:40.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:53:40.725 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:40.726 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:53:40.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:40.971 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:53:41.192 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:41.192 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:53:41.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:41.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:53:41.751 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:41.751 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:53:41.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:41.976 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:53:42.251 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:53:42.252 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:53:42.252 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:53:42.253 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:53:42.253 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:42.253 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:53:42.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:42.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:53:42.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:42.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:53:42.951 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:42.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:53:43.218 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:43.219 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:53:43.493 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:43.493 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:53:43.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:43.787 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:53:44.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:44.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:53:44.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:44.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:53:44.488 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:44.488 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:53:44.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:44.694 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:53:44.986 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:44.986 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:53:45.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:45.211 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:53:45.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:45.611 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:53:45.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:45.836 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:53:46.117 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:46.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:53:46.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:46.336 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:53:46.585 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:46.585 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:53:46.794 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:46.795 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:53:47.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:47.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:53:47.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:47.397 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:53:47.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:47.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:53:47.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:47.887 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:53:48.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:48.137 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:53:48.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:48.387 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:53:48.595 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:48.596 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:53:48.846 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:48.846 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:49.046 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:53:49.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:49.384 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:53:49.621 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:49.621 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:53:49.877 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:49.877 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:53:50.146 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:50.146 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:53:50.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:50.396 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:53:50.671 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:50.671 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:53:50.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:50.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:53:51.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:51.438 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:53:51.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:51.697 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:53:51.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:51.905 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:53:52.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:52.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:53:52.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:52.355 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:53:52.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:52.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:53:52.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:52.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:53:53.105 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:53.106 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:53:53.339 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:53.339 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:53:53.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:53.586 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:53:53.831 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:53.831 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:53:54.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:54.087 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:53:54.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:54.306 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:53:54.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:53:54.555 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:53:54.781 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:53:54.782 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:53:54.783 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:53:54.783 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:53:54.783 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:53:54.783 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:53:54.783 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:53:54.784 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:53:54.784 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:53:54.784 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:53:55.133 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:53:55.133 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:53:55.133 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:53:55.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:53:55.482 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:53:55.803 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:53:55.804 INFO:tasks.workunit.client.0.smithi142.stdout:7.0 41 0 0 0 0 427819008 0 0 22 22 active+clean 2020-03-11T22:53:02.951710+0000 91'122 91:166 [1] 1 [1] 1 0'0 2020-03-11T22:52:35.841059+0000 0'0 2020-03-11T22:52:35.841059+0000 0 2020-03-11T22:53:55.804 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:24.187916+0000 91'150 91:211 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:52:21.881202+0000 0'0 2020-03-11T22:52:21.881202+0000 0 2020-03-11T22:53:55.804 INFO:tasks.workunit.client.0.smithi142.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:53:02.175598+0000 91'150 91:209 [1] 1 [1] 1 0'0 2020-03-11T22:52:26.720637+0000 0'0 2020-03-11T22:52:26.720637+0000 0 2020-03-11T22:53:55.804 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:53:55.804 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:53:55.805 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:53:55.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:53:55.814 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:53:56.402 INFO:tasks.workunit.client.0.smithi142.stderr:nobackfill is set 2020-03-11T22:53:56.410 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:53:57.087 INFO:tasks.workunit.client.0.smithi142.stderr:noout is set 2020-03-11T22:53:57.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:53:57.730 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 7 size to 2 2020-03-11T22:53:57.738 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:53:59.740 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:59.741 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:59.741 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:59.741 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:59.741 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:59.742 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:53:59.742 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:53:59.742 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:59.743 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:59.838 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:53:59.838 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:53:59.838 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:53:59.838 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:53:59.838 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:53:59.839 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:53:59.839 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:53:59.839 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:53:59.839 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:53:59.839 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:53:59.839 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:53:59.839 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:53:59.840 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:53:59.846 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:53:59.846 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:53:59.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:53:59.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:54:00.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-03-11T22:54:00.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:54:02.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:54:02.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:54:02.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:54:02.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 2 = 10 ']' 2020-03-11T22:54:02.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-03-11T22:54:04.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:54:04.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:54:04.535 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:54:04.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:54:04.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:54:04.886 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:04.886 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:05.232 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:05.232 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:54:05.232 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:54:05.232 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:54:05.233 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:54:05.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:05.233 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:05.233 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:05.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836513 2020-03-11T22:54:05.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513' 2020-03-11T22:54:05.358 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:05.358 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672991 2020-03-11T22:54:05.483 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672991' 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:05.484 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:05.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509471 2020-03-11T22:54:05.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672991 2-64424509471' 2020-03-11T22:54:05.610 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:05.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:05.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345949 2020-03-11T22:54:05.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672991 2-64424509471 3-85899345949' 2020-03-11T22:54:05.735 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:05.736 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:05.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182427 2020-03-11T22:54:05.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836513 1-42949672991 2-64424509471 3-85899345949 4-107374182427' 2020-03-11T22:54:05.860 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:05.861 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836513 2020-03-11T22:54:05.861 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:05.861 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:05.862 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836513 2020-03-11T22:54:05.862 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:05.862 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836513 2020-03-11T22:54:05.863 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2020-03-11T22:54:05.863 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:05.863 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836513 2020-03-11T22:54:06.208 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836511 -lt 21474836513 2020-03-11T22:54:06.208 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:07.209 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:07.209 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:07.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836514 -lt 21474836513 2020-03-11T22:54:07.557 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:07.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672991 2020-03-11T22:54:07.558 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:07.558 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:07.559 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672991 2020-03-11T22:54:07.559 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:07.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672991 2020-03-11T22:54:07.560 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2020-03-11T22:54:07.560 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672991 2020-03-11T22:54:07.560 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:07.906 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672991 -lt 42949672991 2020-03-11T22:54:07.907 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:07.907 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509471 2020-03-11T22:54:07.907 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:07.908 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:07.908 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509471 2020-03-11T22:54:07.908 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:07.909 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509471 2020-03-11T22:54:07.909 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2020-03-11T22:54:07.910 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:07.910 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509471 2020-03-11T22:54:08.255 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509471 -lt 64424509471 2020-03-11T22:54:08.255 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:08.256 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345949 2020-03-11T22:54:08.256 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:08.256 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:08.257 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345949 2020-03-11T22:54:08.257 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:08.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345949 2020-03-11T22:54:08.258 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345949' 2020-03-11T22:54:08.258 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:08.258 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345949 2020-03-11T22:54:08.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345949 -lt 85899345949 2020-03-11T22:54:08.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:08.604 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182427 2020-03-11T22:54:08.604 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:08.604 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:08.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182427 2020-03-11T22:54:08.605 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:08.606 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182427 2020-03-11T22:54:08.606 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2020-03-11T22:54:08.606 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:08.606 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182427 2020-03-11T22:54:08.952 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182428 -lt 107374182427 2020-03-11T22:54:08.952 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:08.952 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:08.952 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:08.952 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:08.952 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:08.953 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:08.953 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:08.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:08.953 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:09.039 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:09.039 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:09.039 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:09.039 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:09.039 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:09.039 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:09.040 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:09.041 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:09.041 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:09.041 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:09.041 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:09.041 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:09.041 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:09.045 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:54:09.532 INFO:tasks.workunit.client.0.smithi142.stderr:osd.0 is already out. 2020-03-11T22:54:09.540 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:54:11.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:54:11.542 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:11.542 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:11.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:11.897 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:54:11.897 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:54:11.897 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:54:11.897 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:54:11.897 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:11.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:11.898 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:12.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836517 2020-03-11T22:54:12.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517' 2020-03-11T22:54:12.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:12.023 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:12.149 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672994 2020-03-11T22:54:12.150 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672994' 2020-03-11T22:54:12.150 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:12.150 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:12.276 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509474 2020-03-11T22:54:12.276 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672994 2-64424509474' 2020-03-11T22:54:12.276 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:12.277 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:12.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345952 2020-03-11T22:54:12.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672994 2-64424509474 3-85899345952' 2020-03-11T22:54:12.405 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:12.405 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:12.549 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182431 2020-03-11T22:54:12.549 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836517 1-42949672994 2-64424509474 3-85899345952 4-107374182431' 2020-03-11T22:54:12.549 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:12.550 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836517 2020-03-11T22:54:12.550 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:12.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:12.550 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836517 2020-03-11T22:54:12.550 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:12.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836517 2020-03-11T22:54:12.550 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2020-03-11T22:54:12.551 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:12.551 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836517 2020-03-11T22:54:12.896 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836517 2020-03-11T22:54:12.896 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:13.898 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:13.898 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:14.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836517 2020-03-11T22:54:14.249 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:15.250 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:54:15.250 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:15.605 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836517 2020-03-11T22:54:15.606 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:16.606 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 298 -eq 0 ']' 2020-03-11T22:54:16.607 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:16.961 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836517 2020-03-11T22:54:16.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:17.962 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 297 -eq 0 ']' 2020-03-11T22:54:17.963 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:18.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836518 -lt 21474836517 2020-03-11T22:54:18.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:18.315 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672994 2020-03-11T22:54:18.315 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:18.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:18.316 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672994 2020-03-11T22:54:18.316 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:18.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672994 2020-03-11T22:54:18.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2020-03-11T22:54:18.318 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:18.318 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672994 2020-03-11T22:54:18.663 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672995 -lt 42949672994 2020-03-11T22:54:18.664 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:18.664 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509474 2020-03-11T22:54:18.664 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:18.664 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:18.665 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509474 2020-03-11T22:54:18.665 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:18.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509474 2020-03-11T22:54:18.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2020-03-11T22:54:18.666 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:18.667 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509474 2020-03-11T22:54:19.009 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509475 -lt 64424509474 2020-03-11T22:54:19.010 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:19.011 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345952 2020-03-11T22:54:19.011 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:19.011 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:19.012 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345952 2020-03-11T22:54:19.013 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:19.013 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345952 2020-03-11T22:54:19.013 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2020-03-11T22:54:19.013 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:19.014 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345952 2020-03-11T22:54:19.357 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345953 -lt 85899345952 2020-03-11T22:54:19.357 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:19.358 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182431 2020-03-11T22:54:19.358 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:19.359 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:19.359 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182431 2020-03-11T22:54:19.359 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:19.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182431 2020-03-11T22:54:19.360 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2020-03-11T22:54:19.360 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:19.360 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182431 2020-03-11T22:54:19.703 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182432 -lt 107374182431 2020-03-11T22:54:19.705 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:19.705 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:19.705 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:19.705 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:19.705 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:19.705 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:19.705 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:19.706 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:19.706 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:19.792 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:19.792 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:19.793 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:19.793 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:19.793 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:19.793 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:19.794 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:19.794 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:19.794 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:19.794 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:19.794 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:19.795 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:19.795 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:19.795 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:19.795 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:19.796 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:19.796 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:19.796 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:19.796 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:19.797 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:19.797 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:19.797 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:19.797 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:19.798 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:19.798 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:19.798 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:19.798 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:19.799 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:19.799 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:19.799 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:19.799 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:19.800 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:54:20.129 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:54:20.129 INFO:tasks.workunit.client.0.smithi142.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:54:04.781487+0000 91'150 101:211 [1,4] 1 [1] 1 0'0 2020-03-11T22:52:35.841059+0000 0'0 2020-03-11T22:52:35.841059+0000 0 2020-03-11T22:54:20.129 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:54:12.628660+0000 91'150 102:226 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:52:21.881202+0000 0'0 2020-03-11T22:52:21.881202+0000 0 2020-03-11T22:54:20.129 INFO:tasks.workunit.client.0.smithi142.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:53:02.175598+0000 91'150 101:216 [1] 1 [1] 1 0'0 2020-03-11T22:52:26.720637+0000 0'0 2020-03-11T22:52:26.720637+0000 0 2020-03-11T22:54:20.129 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:54:20.130 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:54:20.130 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:54:20.138 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:54:20.747 INFO:tasks.workunit.client.0.smithi142.stderr:set pool 3 size to 2 2020-03-11T22:54:20.756 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:54:22.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:54:22.757 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:22.757 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:23.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:23.115 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:54:23.115 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:54:23.115 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:54:23.115 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:54:23.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:23.116 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:23.116 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:23.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836521 2020-03-11T22:54:23.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521' 2020-03-11T22:54:23.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:23.241 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:23.375 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672999 2020-03-11T22:54:23.375 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949672999' 2020-03-11T22:54:23.375 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:23.375 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:23.501 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509478 2020-03-11T22:54:23.501 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949672999 2-64424509478' 2020-03-11T22:54:23.501 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:23.502 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:23.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345956 2020-03-11T22:54:23.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949672999 2-64424509478 3-85899345956' 2020-03-11T22:54:23.627 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:23.627 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:23.752 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182435 2020-03-11T22:54:23.752 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836521 1-42949672999 2-64424509478 3-85899345956 4-107374182435' 2020-03-11T22:54:23.752 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:23.753 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836521 2020-03-11T22:54:23.753 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:23.754 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:23.754 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836521 2020-03-11T22:54:23.754 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:23.755 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836521 2020-03-11T22:54:23.755 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2020-03-11T22:54:23.755 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:23.756 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836521 2020-03-11T22:54:24.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836521 2020-03-11T22:54:24.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:25.100 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:25.101 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:25.457 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836521 -lt 21474836521 2020-03-11T22:54:25.458 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:25.458 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672999 2020-03-11T22:54:25.458 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:25.458 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:25.459 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672999 2020-03-11T22:54:25.459 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:25.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672999 2020-03-11T22:54:25.460 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2020-03-11T22:54:25.460 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:25.461 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949672999 2020-03-11T22:54:25.810 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672999 -lt 42949672999 2020-03-11T22:54:25.810 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:25.811 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509478 2020-03-11T22:54:25.811 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:25.811 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:25.812 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509478 2020-03-11T22:54:25.812 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:25.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509478 2020-03-11T22:54:25.813 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2020-03-11T22:54:25.813 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:25.813 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509478 2020-03-11T22:54:26.177 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509479 -lt 64424509478 2020-03-11T22:54:26.177 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:26.178 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345956 2020-03-11T22:54:26.178 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:26.178 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:26.179 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345956 2020-03-11T22:54:26.179 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:26.180 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345956 2020-03-11T22:54:26.180 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345956' 2020-03-11T22:54:26.180 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:26.180 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345956 2020-03-11T22:54:26.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345957 -lt 85899345956 2020-03-11T22:54:26.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:26.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182435 2020-03-11T22:54:26.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:26.529 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:26.529 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182435 2020-03-11T22:54:26.529 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:26.530 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182435 2020-03-11T22:54:26.530 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182435' 2020-03-11T22:54:26.531 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:26.531 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182435 2020-03-11T22:54:26.878 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182435 -lt 107374182435 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:26.974 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:26.974 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:26.974 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:26.975 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:26.975 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:26.975 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:26.975 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:26.975 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:26.975 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:26.975 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:26.976 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:54:26.977 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:26.977 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:26.977 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:26.977 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:26.977 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:26.977 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:26.977 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:26.978 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:26.978 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:26.978 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:26.978 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:26.978 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:26.978 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:26.978 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:26.979 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:26.980 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:26.980 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:26.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:54:27.298 INFO:tasks.workunit.client.0.smithi142.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:54:27.298 INFO:tasks.workunit.client.0.smithi142.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:54:04.781487+0000 91'150 105:214 [1,4] 1 [1] 1 0'0 2020-03-11T22:52:35.841059+0000 0'0 2020-03-11T22:52:35.841059+0000 0 2020-03-11T22:54:27.298 INFO:tasks.workunit.client.0.smithi142.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:54:12.628660+0000 91'150 105:229 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:52:21.881202+0000 0'0 2020-03-11T22:52:21.881202+0000 0 2020-03-11T22:54:27.298 INFO:tasks.workunit.client.0.smithi142.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-03-11T22:54:22.535802+0000 91'150 106:226 [1,2] 1 [1] 1 0'0 2020-03-11T22:52:26.720637+0000 0'0 2020-03-11T22:52:26.720637+0000 0 2020-03-11T22:54:27.298 INFO:tasks.workunit.client.0.smithi142.stdout: 2020-03-11T22:54:27.299 INFO:tasks.workunit.client.0.smithi142.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:54:27.299 INFO:tasks.workunit.client.0.smithi142.stderr:dumped pgs 2020-03-11T22:54:27.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:27.308 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:54:27.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:54:27.310 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:54:27.311 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:27.311 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:54:27.313 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:54:27.313 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:54:27.313 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:54:27.314 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:27.314 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:54:27.315 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:54:27.316 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:54:27.316 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:54:27.317 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:54:27.317 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:54:27.318 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:54:27.651 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:54:27.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:54:29.654 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:29.654 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:29.654 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:29.654 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:29.654 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:29.654 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:29.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:29.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:29.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:29.760 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:29.760 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:29.761 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:29.761 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:29.761 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:29.761 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:29.761 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:29.761 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:29.762 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:29.762 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:29.762 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:29.762 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:29.762 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:29.762 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:29.762 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 254, 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:29.763 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:29.764 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:29.764 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:29.764 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:29.764 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:29.764 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:29.764 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:29.764 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:29.765 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:29.765 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:29.765 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:29.765 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:29.765 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:29.765 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:29.766 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:29.766 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:29.766 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:29.766 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:29.766 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:29.766 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:29.768 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:29.768 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:54:29.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:54:29.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:54:29.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:54:29.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:29.769 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:30.114 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:30.114 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:54:30.114 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:54:30.114 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:54:30.114 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:54:30.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:30.115 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:30.115 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:30.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836524 2020-03-11T22:54:30.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524' 2020-03-11T22:54:30.241 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:30.241 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:30.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673002 2020-03-11T22:54:30.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2020-03-11T22:54:30.369 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:30.369 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:30.494 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509482 2020-03-11T22:54:30.495 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509482' 2020-03-11T22:54:30.495 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:30.495 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:30.620 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345960 2020-03-11T22:54:30.621 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509482 3-85899345960' 2020-03-11T22:54:30.621 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:30.621 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:30.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182438 2020-03-11T22:54:30.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509482 3-85899345960 4-107374182438' 2020-03-11T22:54:30.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:30.747 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836524 2020-03-11T22:54:30.747 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:30.747 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:30.748 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836524 2020-03-11T22:54:30.748 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:30.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836524 2020-03-11T22:54:30.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2020-03-11T22:54:30.749 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:30.749 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836524 2020-03-11T22:54:31.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836522 -lt 21474836524 2020-03-11T22:54:31.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:32.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:32.097 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:32.450 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836524 -lt 21474836524 2020-03-11T22:54:32.450 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:32.450 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673002 2020-03-11T22:54:32.450 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:32.451 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:32.451 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673002 2020-03-11T22:54:32.452 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:32.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673002 2020-03-11T22:54:32.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2020-03-11T22:54:32.453 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:32.453 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949673002 2020-03-11T22:54:32.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673002 -lt 42949673002 2020-03-11T22:54:32.804 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:32.805 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509482 2020-03-11T22:54:32.805 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:32.806 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:32.806 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509482 2020-03-11T22:54:32.806 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:32.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509482 2020-03-11T22:54:32.807 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2020-03-11T22:54:32.807 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:32.808 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509482 2020-03-11T22:54:33.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509482 -lt 64424509482 2020-03-11T22:54:33.154 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:33.155 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345960 2020-03-11T22:54:33.155 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:33.155 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:33.156 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345960 2020-03-11T22:54:33.156 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:33.157 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345960 2020-03-11T22:54:33.157 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345960' 2020-03-11T22:54:33.158 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:33.158 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345960 2020-03-11T22:54:33.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345960 -lt 85899345960 2020-03-11T22:54:33.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:33.506 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182438 2020-03-11T22:54:33.506 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:33.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:33.507 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182438 2020-03-11T22:54:33.507 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:33.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182438 2020-03-11T22:54:33.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182438' 2020-03-11T22:54:33.508 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:33.508 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182438 2020-03-11T22:54:33.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182439 -lt 107374182438 2020-03-11T22:54:33.856 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:54:34.198 INFO:tasks.workunit.client.0.smithi142.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:54:34.198 INFO:tasks.workunit.client.0.smithi142.stderr: 2020-03-11T22:54:34.206 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:54:36.208 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:36.209 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:36.209 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:36.209 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:36.209 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:36.209 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:36.209 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:36.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:36.210 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:36.314 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:36.315 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:36.315 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:36.315 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:36.315 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:36.316 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:36.316 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:36.316 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:36.316 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:36.316 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:36.316 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:36.316 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:36.317 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:36.317 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:36.317 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:36.317 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:36.317 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:36.317 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:54:36.317 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:36.318 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:36.318 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:36.318 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:36.318 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:36.318 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:36.318 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:36.318 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:36.319 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:36.319 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:36.319 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:36.319 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:36.319 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:36.319 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:36.319 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:36.320 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:36.321 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:36.322 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:36.322 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:54:36.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:54:36.322 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:54:36.323 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:36.323 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:54:36.323 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:54:36.323 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:54:36.323 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:54:36.323 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:36.323 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:54:36.324 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:54:36.324 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:54:36.325 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:54:36.652 INFO:tasks.workunit.client.0.smithi142.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:54:36.652 INFO:tasks.workunit.client.0.smithi142.stderr: 2020-03-11T22:54:36.660 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:54:41.662 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:41.662 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:41.662 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:41.665 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:41.665 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:41.666 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:41.666 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:41.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:41.666 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:41.760 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:41.760 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:41.761 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:41.761 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:41.761 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:41.761 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:41.762 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:41.762 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:41.762 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:41.762 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:41.762 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:41.762 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:41.763 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:41.763 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:41.763 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:41.763 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:41.763 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:54:41.763 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:41.763 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:41.764 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:41.765 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:41.765 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:41.765 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:41.765 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:41.765 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:41.765 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:41.766 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:41.766 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:41.766 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:41.766 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:41.766 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:41.767 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:41.767 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:41.767 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:41.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:54:41.768 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:41.769 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:42.118 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:42.118 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:54:42.118 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:54:42.118 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:54:42.118 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:54:42.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:42.119 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:42.119 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:42.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836529 2020-03-11T22:54:42.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529' 2020-03-11T22:54:42.248 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:42.248 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:42.373 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673006 2020-03-11T22:54:42.373 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673006' 2020-03-11T22:54:42.373 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:42.374 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:42.498 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509486 2020-03-11T22:54:42.498 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509486' 2020-03-11T22:54:42.499 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:42.499 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:42.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345964 2020-03-11T22:54:42.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509486 3-85899345964' 2020-03-11T22:54:42.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:42.625 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:42.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182443 2020-03-11T22:54:42.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509486 3-85899345964 4-107374182443' 2020-03-11T22:54:42.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:42.751 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836529 2020-03-11T22:54:42.751 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:42.752 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:42.752 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836529 2020-03-11T22:54:42.752 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:42.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836529 2020-03-11T22:54:42.753 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2020-03-11T22:54:42.753 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:42.753 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836529 2020-03-11T22:54:43.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836526 -lt 21474836529 2020-03-11T22:54:43.096 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:44.097 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:44.097 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:44.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836529 -lt 21474836529 2020-03-11T22:54:44.452 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:44.452 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673006 2020-03-11T22:54:44.453 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:44.453 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:44.453 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673006 2020-03-11T22:54:44.454 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:44.454 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673006 2020-03-11T22:54:44.455 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2020-03-11T22:54:44.455 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:44.455 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949673006 2020-03-11T22:54:44.824 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673007 -lt 42949673006 2020-03-11T22:54:44.824 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:44.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509486 2020-03-11T22:54:44.825 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:44.826 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:44.826 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509486 2020-03-11T22:54:44.826 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:44.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509486 2020-03-11T22:54:44.827 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2020-03-11T22:54:44.827 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:44.827 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509486 2020-03-11T22:54:45.170 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509486 -lt 64424509486 2020-03-11T22:54:45.171 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:45.171 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345964 2020-03-11T22:54:45.171 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:45.172 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:45.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345964 2020-03-11T22:54:45.172 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:45.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345964 2020-03-11T22:54:45.173 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345964' 2020-03-11T22:54:45.173 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:45.173 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345964 2020-03-11T22:54:45.526 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345965 -lt 85899345964 2020-03-11T22:54:45.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:45.527 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182443 2020-03-11T22:54:45.527 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:45.528 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:45.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182443 2020-03-11T22:54:45.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:45.529 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182443 2020-03-11T22:54:45.529 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182443' 2020-03-11T22:54:45.529 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:45.529 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182443 2020-03-11T22:54:45.873 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182443 -lt 107374182443 2020-03-11T22:54:45.874 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:54:46.220 INFO:tasks.workunit.client.0.smithi142.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:54:46.220 INFO:tasks.workunit.client.0.smithi142.stderr: 2020-03-11T22:54:46.227 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:54:48.230 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:48.230 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:48.230 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:48.231 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:48.231 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:48.231 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:48.231 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:48.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:54:48.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:48.335 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:48.335 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:48.335 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:48.336 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:48.336 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:48.336 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:48.336 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:48.336 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:48.336 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:48.336 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:48.337 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:48.337 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:48.337 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:48.337 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:48.337 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:48.337 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:48.337 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:48.339 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:54:48.339 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:48.339 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:48.339 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:48.339 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:48.339 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:48.339 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:48.340 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:48.341 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:48.341 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:48.341 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:48.341 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:48.341 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:48.341 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:48.341 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:48.342 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:48.342 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:48.342 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:48.343 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:48.343 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:54:48.343 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:54:48.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:54:48.344 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:48.344 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:54:48.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:54:48.344 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:54:48.344 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:54:48.344 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:54:48.345 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:54:48.345 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:54:48.346 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:54:48.346 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:54:48.954 INFO:tasks.workunit.client.0.smithi142.stderr:noout is unset 2020-03-11T22:54:48.963 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:54:49.747 INFO:tasks.workunit.client.0.smithi142.stderr:nobackfill is unset 2020-03-11T22:54:49.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:54:49.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:54:49.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:54:49.757 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:49.757 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:49.757 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55987 2020-03-11T22:54:49.757 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55987/ceph-osd.1.asok 2020-03-11T22:54:49.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:54:49.758 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:54:49.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:49.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:49.759 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:49.759 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:49.759 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:49.759 INFO:tasks.workunit.client.0.smithi142.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:49.759 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:49.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:49.760 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:49.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:49.847 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:49.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:49.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:49.848 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:50.200 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:50.200 INFO:tasks.workunit.client.0.smithi142.stderr:1 2020-03-11T22:54:50.200 INFO:tasks.workunit.client.0.smithi142.stderr:2 2020-03-11T22:54:50.200 INFO:tasks.workunit.client.0.smithi142.stderr:3 2020-03-11T22:54:50.201 INFO:tasks.workunit.client.0.smithi142.stderr:4' 2020-03-11T22:54:50.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:50.201 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:50.201 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:50.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836532 2020-03-11T22:54:50.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532' 2020-03-11T22:54:50.329 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:50.330 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:50.457 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949673010 2020-03-11T22:54:50.457 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673010' 2020-03-11T22:54:50.457 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:50.458 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:50.583 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509490 2020-03-11T22:54:50.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509490' 2020-03-11T22:54:50.584 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:50.584 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:50.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345968 2020-03-11T22:54:50.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509490 3-85899345968' 2020-03-11T22:54:50.713 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:50.714 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:50.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182446 2020-03-11T22:54:50.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509490 3-85899345968 4-107374182446' 2020-03-11T22:54:50.841 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:50.842 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836532 2020-03-11T22:54:50.842 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:50.842 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:50.843 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836532 2020-03-11T22:54:50.843 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:50.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836532 2020-03-11T22:54:50.844 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2020-03-11T22:54:50.844 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:50.844 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.0 seq 21474836532 2020-03-11T22:54:51.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836532 2020-03-11T22:54:51.215 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:52.217 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:52.218 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:52.644 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836532 -lt 21474836532 2020-03-11T22:54:52.645 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:52.645 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949673010 2020-03-11T22:54:52.645 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:52.646 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:52.646 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949673010 2020-03-11T22:54:52.646 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:52.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949673010 2020-03-11T22:54:52.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2020-03-11T22:54:52.647 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:52.648 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.1 seq 42949673010 2020-03-11T22:54:53.062 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949673010 -lt 42949673010 2020-03-11T22:54:53.062 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:53.063 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509490 2020-03-11T22:54:53.063 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:53.064 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:53.064 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509490 2020-03-11T22:54:53.064 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:53.065 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509490 2020-03-11T22:54:53.065 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2020-03-11T22:54:53.066 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:53.066 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.2 seq 64424509490 2020-03-11T22:54:53.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509490 -lt 64424509490 2020-03-11T22:54:53.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:53.505 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-85899345968 2020-03-11T22:54:53.506 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:53.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:53.506 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-85899345968 2020-03-11T22:54:53.507 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:53.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=85899345968 2020-03-11T22:54:53.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2020-03-11T22:54:53.508 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:53.508 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.3 seq 85899345968 2020-03-11T22:54:53.865 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 85899345968 -lt 85899345968 2020-03-11T22:54:53.865 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:53.866 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-107374182446 2020-03-11T22:54:53.866 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:53.866 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:53.867 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-107374182446 2020-03-11T22:54:53.867 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:53.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=107374182446 2020-03-11T22:54:53.868 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2020-03-11T22:54:53.868 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:53.869 INFO:tasks.workunit.client.0.smithi142.stdout:waiting osd.4 seq 107374182446 2020-03-11T22:54:54.220 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 107374182447 -lt 107374182446 2020-03-11T22:54:54.220 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:54:54.220 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:54.221 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:54.674 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:54:54.674 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:54.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:54.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:54.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:54.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:54.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:54.675 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:55.099 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:55.099 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:55.100 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:55.100 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:55.553 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:55.553 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:54:55.553 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:54:55.553 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:54:55.553 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:55.554 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:55.554 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:55.640 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:55.640 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:55.640 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:55.640 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:55.640 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:55.641 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:55.641 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:55.641 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:55.641 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:55.641 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:55.641 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:55.641 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:55.642 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:55.642 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:55.642 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:55.642 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:55.642 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:54:55.642 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:55.642 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:55.643 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:55.644 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:55.644 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:55.644 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:55.644 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:55.644 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:55.644 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:55.644 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:55.645 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:55.645 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:55.645 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:55.645 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:55.645 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:55.645 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:55.645 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:55.647 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:55.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:55.749 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:55.749 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:55.749 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:55.749 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:55.750 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:55.750 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:55.750 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:56.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:56.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:56.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:56.086 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:56.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:56.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:56.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:56.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:56.533 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:56.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:56.534 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:56.534 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:56.534 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:56.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=10047058 2020-03-11T22:54:56.991 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 10047058 '!=' null 2020-03-11T22:54:56.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:56.992 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:56.992 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:56.992 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:57.078 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:57.078 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:57.078 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:57.078 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:57.079 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:57.079 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:57.079 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:57.079 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:57.079 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:57.079 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:57.079 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:57.080 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:57.081 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:57.081 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:57.081 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:57.081 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:57.081 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:57.081 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:57.082 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:57.082 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:57.082 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:57.082 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:57.082 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:57.082 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:57.082 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:57.083 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:57.083 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:57.083 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:57.083 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:57.083 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:57.084 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:57.084 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:57.084 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:57.084 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:57.085 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:57.186 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:57.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:57.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:57.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:57.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:57.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:57.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:57.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:57.524 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:57.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:57.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:57.525 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:57.977 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:57.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:57.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:57.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:57.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:57.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:57.978 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:57.979 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:57.979 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:58.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=7862990 2020-03-11T22:54:58.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 7862990 '!=' null 2020-03-11T22:54:58.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:58.481 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:58.482 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:58.482 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:58.570 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:54:58.570 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:54:58.570 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:58.570 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:58.571 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:54:58.571 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:58.571 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:54:58.571 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:58.571 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:58.571 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:54:58.571 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:54:58.572 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:58.573 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:54:58.573 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:54:58.573 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:58.573 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:58.573 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:58.573 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:58.573 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:54:58.574 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:54:58.574 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:54:58.574 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:54:58.574 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:54:58.574 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:54:58.574 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:58.574 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:54:58.575 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:54:58.575 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:54:58.575 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:54:58.575 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:54:58.575 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:54:58.575 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:54:58.575 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:54:58.576 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:54:58.577 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:58.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:58.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:58.679 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:58.679 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:58.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:58.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:58.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:58.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:59.019 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:59.019 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:59.020 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:59.020 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:59.477 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:54:59.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:59.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:59.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:59.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:59.478 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:59.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:59.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:59.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:59.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20339172 2020-03-11T22:54:59.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20339172 '!=' null 2020-03-11T22:54:59.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:59.927 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:54:59.927 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:54:59.927 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:00.016 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:00.016 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:00.016 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:00.017 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:00.017 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:00.017 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:00.017 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:00.017 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:00.017 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:00.018 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:55:00.019 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:00.019 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:00.019 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:00.019 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:00.019 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:00.019 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:00.020 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:00.020 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:00.020 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:00.020 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:00.020 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:00.020 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:55:00.020 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:00.021 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:00.022 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:00.022 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:00.022 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:00.023 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:00.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:00.124 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:00.125 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:00.125 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:00.126 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:00.126 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:00.126 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:00.126 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:00.467 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:00.468 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:00.468 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:00.468 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:00.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:00.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:00.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:00.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:00.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:00.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:00.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:00.924 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:00.925 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:01.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20339172 2020-03-11T22:55:01.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20339172 '!=' null 2020-03-11T22:55:01.411 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:01.412 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:01.412 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:01.412 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:01.500 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:01.501 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:01.501 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:01.501 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:01.501 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:01.501 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:01.502 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:01.502 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:01.502 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:01.502 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:01.502 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:01.502 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:01.502 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:01.503 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:01.504 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:01.504 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:01.504 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:01.504 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:01.504 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:01.504 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:01.504 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:01.505 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:01.506 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:01.506 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:01.506 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:01.506 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:01.506 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:01.507 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:01.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:01.609 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:01.609 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:01.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:01.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:01.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:01.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:01.610 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:01.989 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:01.990 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:01.990 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:01.990 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:02.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:02.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:02.442 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:02.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:02.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:02.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:02.443 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:02.443 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:02.443 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:02.932 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17369595 2020-03-11T22:55:02.933 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17369595 '!=' null 2020-03-11T22:55:02.933 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:02.933 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:02.933 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:02.933 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:03.021 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:03.022 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:03.022 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:03.022 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:03.022 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:03.022 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:03.023 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:03.023 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:03.023 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:03.023 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:03.023 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:03.023 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:03.023 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:03.024 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:03.024 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:03.024 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:03.024 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:55:03.024 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:03.024 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:03.025 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:03.025 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:03.025 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:03.025 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:03.025 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:03.025 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:03.026 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:03.026 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:03.026 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:03.026 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:55:03.026 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:55:03.027 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:03.027 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:03.027 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:03.027 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:03.027 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:03.028 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:03.028 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:03.028 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:03.028 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:03.028 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:03.028 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:03.029 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:03.130 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:03.131 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:03.131 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:03.131 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:03.131 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:03.131 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:03.131 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:03.132 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:03.471 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:03.472 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:03.472 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:03.472 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:03.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:03.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:03.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:03.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:03.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:03.923 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:03.924 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:03.924 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:03.924 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:04.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957114 2020-03-11T22:55:04.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957114 '!=' null 2020-03-11T22:55:04.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:04.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:04.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:04.480 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:04.567 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:04.568 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:04.568 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:04.568 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:04.568 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:04.568 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:04.568 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:04.569 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:04.569 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:04.569 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:04.569 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:04.569 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:04.569 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:04.569 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:04.570 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:04.570 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:04.570 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:55:04.570 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:04.570 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:04.570 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:04.570 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:04.571 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:04.571 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:04.571 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:04.571 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:04.571 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:04.571 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:04.571 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:04.572 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:55:04.572 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:55:04.572 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:04.572 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:04.573 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:04.573 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:04.573 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:04.573 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:04.573 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:04.573 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:04.574 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:04.574 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:04.574 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:04.574 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:04.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:04.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:04.677 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:04.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:04.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:04.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:04.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:04.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:05.016 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:05.016 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:05.016 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:05.016 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:05.488 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:05.488 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:05.488 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:05.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:05.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:05.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:05.489 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:05.489 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:05.489 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:05.988 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22715166 2020-03-11T22:55:05.988 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22715166 '!=' null 2020-03-11T22:55:05.989 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:05.989 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:05.989 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:05.989 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:06.077 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:06.077 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:06.077 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:06.077 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:06.077 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:06.077 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:06.078 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:06.079 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:06.079 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:06.079 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:55:06.079 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:06.079 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:06.079 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:06.079 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:06.080 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:06.080 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:06.080 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:06.080 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:06.080 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:06.080 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:06.081 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:06.081 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:55:06.081 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:55:06.081 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:06.081 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:06.081 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:06.082 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:06.082 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:06.082 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:06.082 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:06.082 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:06.082 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:06.082 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:06.083 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:06.084 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:06.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:06.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:06.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:06.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:06.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:06.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:06.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:06.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:06.527 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:06.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:06.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:06.528 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:06.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:06.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:06.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:06.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:06.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:06.980 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:06.981 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:06.981 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:06.981 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:07.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22715093 2020-03-11T22:55:07.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22715093 '!=' null 2020-03-11T22:55:07.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:07.479 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:07.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:07.479 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:07.567 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:07.567 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:07.567 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:07.568 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:07.568 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:07.568 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:07.568 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:07.568 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:07.568 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:07.568 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:07.569 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:07.569 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:07.569 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:07.569 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:07.569 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:07.569 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:07.569 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:55:07.570 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:07.570 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:07.570 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:07.570 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:07.570 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:07.570 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:07.570 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:07.571 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:07.571 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:07.571 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:07.571 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:07.571 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:55:07.571 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:55:07.571 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:07.572 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:07.572 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:07.572 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:07.572 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:07.572 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:07.572 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:07.572 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:07.573 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:07.573 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:07.573 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:07.575 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:07.676 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:07.677 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:07.677 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:07.677 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:07.677 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:07.677 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:07.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:07.678 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:08.011 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:08.011 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:08.011 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:08.012 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:08.469 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:08.469 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:08.469 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:08.469 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:08.469 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:08.469 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:08.470 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:08.470 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:08.470 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:08.964 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22715093 2020-03-11T22:55:08.965 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22715093 '!=' null 2020-03-11T22:55:08.965 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:08.965 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:08.965 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:08.965 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:09.053 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:09.053 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:09.054 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:09.054 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:09.054 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:09.054 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:09.054 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:09.054 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:09.054 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:09.055 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:09.055 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:09.055 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:09.055 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:09.055 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:09.055 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:09.055 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 151, 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:09.056 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:09.057 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:09.057 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:09.057 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:09.057 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:09.057 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "7.0", 2020-03-11T22:55:09.057 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 254, 2020-03-11T22:55:09.057 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:09.058 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:09.059 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:09.059 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:09.060 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:09.161 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:09.162 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:09.162 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:09.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:09.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:09.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:09.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:09.163 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:09.514 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:09.514 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:09.515 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:09.518 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:10.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:10.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:10.066 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:10.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:10.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:10.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:10.067 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:10.067 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:10.067 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:10.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34072332 2020-03-11T22:55:10.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34072332 '!=' null 2020-03-11T22:55:10.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:10.626 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:10.626 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:10.626 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:10.714 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:10.715 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:10.716 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:10.716 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:10.716 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:10.716 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:10.716 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:10.716 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:10.716 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:10.717 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:10.717 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:10.717 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:10.717 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:10.717 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:10.717 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:10.717 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:10.718 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:10.718 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:10.718 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:10.718 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:10.718 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:10.718 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:10.721 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:10.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:10.822 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:10.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:10.823 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:10.824 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:10.824 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:10.824 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:10.824 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:11.160 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:11.160 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:11.160 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:11.161 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:11.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:11.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:11.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:11.612 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:11.613 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:11.613 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:11.613 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:11.613 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:11.613 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:12.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036128 2020-03-11T22:55:12.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036128 '!=' null 2020-03-11T22:55:12.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:12.143 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:12.143 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:12.144 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:12.231 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:12.231 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:12.232 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:12.232 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:12.232 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:12.232 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:12.232 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:12.232 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:12.232 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:12.233 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:12.233 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:12.233 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:12.233 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:12.233 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:12.233 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:12.233 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:12.234 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:12.234 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:12.234 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:12.234 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:12.234 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:12.234 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:12.235 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:12.235 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:12.235 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:12.235 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:12.236 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:12.237 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:12.237 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:12.237 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:12.238 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:12.239 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:12.340 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:12.340 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:12.340 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:12.340 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:12.340 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:12.341 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:12.679 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:55:12.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:12.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:12.680 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:13.131 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:55:13.131 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:55:13.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:13.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:13.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:13.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:13.132 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:13.132 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:13.132 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:13.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32761663 2020-03-11T22:55:13.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32761663 '!=' null 2020-03-11T22:55:13.654 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:13.655 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:13.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:13.655 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:13.743 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:13.743 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:13.743 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:13.743 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:13.744 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:13.745 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:13.745 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:13.745 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:13.745 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:13.745 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:13.745 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:13.745 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:13.746 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:13.746 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:13.746 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:13.746 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:13.746 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:13.746 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:13.746 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:13.747 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:13.747 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:13.747 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:13.747 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:13.747 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:13.750 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:13.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:13.851 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:13.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:13.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:13.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:13.852 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:13.853 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:13.853 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:14.194 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:14.194 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:14.195 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:14.195 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:14.651 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:14.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:55:14.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:55:14.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:55:14.652 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:14.652 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:14.652 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:14.739 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:14.739 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:14.739 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:14.739 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:14.739 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:14.740 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:14.740 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:14.740 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:14.740 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:14.740 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:14.740 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:14.740 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:14.741 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:14.741 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:14.741 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:14.741 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:14.742 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:14.742 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:14.742 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:14.742 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:14.742 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:14.742 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:14.742 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:14.743 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:14.743 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:14.743 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:14.743 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:14.743 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:14.743 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:14.743 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:14.744 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:14.746 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:14.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:14.848 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:14.848 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:14.848 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:14.848 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:14.849 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:14.849 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:14.849 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:15.187 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:15.187 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:15.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:15.188 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:55:15.641 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:15.642 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:16.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32761594 2020-03-11T22:55:16.094 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32761594 '!=' null 2020-03-11T22:55:16.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:16.095 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:16.095 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:16.095 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:16.182 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:16.182 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:16.182 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:16.183 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:16.184 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:16.184 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:16.184 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:16.184 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:16.184 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:16.184 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:16.184 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:16.185 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:16.185 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:16.185 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:16.185 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:16.185 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:16.185 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:16.186 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:16.186 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:16.186 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:16.186 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:16.186 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:16.186 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:16.186 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:16.188 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:16.293 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:16.294 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:16.294 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:16.294 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:16.294 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:16.294 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:16.294 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:16.295 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:16.628 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:16.628 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:16.629 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:16.629 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:55:17.111 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:17.112 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:17.112 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:17.112 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:17.558 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725834 2020-03-11T22:55:17.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725834 '!=' null 2020-03-11T22:55:17.624 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:17.625 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:17.625 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:17.625 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:17.646 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:17.646 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:17.647 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:17.647 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:17.647 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:17.647 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:17.647 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:17.647 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:17.647 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:17.648 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:17.649 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:17.649 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:17.649 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:17.649 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:17.649 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:17.649 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:17.649 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:17.650 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:17.650 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:17.650 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:17.650 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:17.650 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:17.650 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:17.650 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:17.653 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:17.754 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:17.755 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:17.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:17.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:17.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:17.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:17.756 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:17.757 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:18.092 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:18.092 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:18.092 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:18.093 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:18.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:18.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:55:18.578 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:18.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:18.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:18.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:18.579 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:18.579 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:18.579 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:19.036 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15725834 2020-03-11T22:55:19.036 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15725834 '!=' null 2020-03-11T22:55:19.036 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:19.036 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:19.037 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:19.037 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:19.123 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:19.123 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:19.123 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:19.124 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:19.124 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:19.124 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:19.124 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:19.124 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:19.124 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:19.124 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:19.125 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:19.125 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:19.125 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:19.125 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:19.125 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:19.125 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:19.125 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:19.126 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:19.126 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:19.126 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:19.126 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:19.127 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:19.127 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:19.127 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:19.127 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:19.127 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:19.127 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:19.127 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:19.128 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:19.128 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:19.128 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:19.130 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:19.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:19.231 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:19.232 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:19.232 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:19.232 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:19.233 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:19.233 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:19.233 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:19.618 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:19.618 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:19.619 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:19.619 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:20.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:20.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:55:20.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:20.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:20.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:20.068 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:20.069 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:20.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:20.069 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:20.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26682356 2020-03-11T22:55:20.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26682356 '!=' null 2020-03-11T22:55:20.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:20.522 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:20.522 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:20.523 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:20.610 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:20.610 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:20.610 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:20.611 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:20.611 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:20.611 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:20.611 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:20.611 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:20.611 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:20.611 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:20.612 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:20.612 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:20.612 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:20.612 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:20.612 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:20.613 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:20.613 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:20.613 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:20.613 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:20.613 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:20.614 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:20.614 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:20.614 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:20.614 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:20.614 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:20.615 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:20.615 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:20.615 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:20.615 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:20.615 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:20.616 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:20.617 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:20.718 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:20.719 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:20.719 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:20.720 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:20.720 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:20.720 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:20.720 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:20.720 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:21.056 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:21.056 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:21.057 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:21.057 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:21.508 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:21.509 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:55:21.509 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:21.509 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:21.509 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:21.509 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:21.509 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:21.509 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:21.510 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:22.008 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22998485 2020-03-11T22:55:22.008 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22998485 '!=' null 2020-03-11T22:55:22.008 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:22.008 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:22.008 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:22.009 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:22.095 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:22.095 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:22.096 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:22.096 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:22.096 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:22.096 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:22.096 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:22.096 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:22.096 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:22.097 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:22.098 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:22.098 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:22.098 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:22.098 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:22.098 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:22.098 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:22.098 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:22.099 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:22.099 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:22.099 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:22.099 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:22.099 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:22.100 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:22.100 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:22.102 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:22.204 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:22.205 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:22.205 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:22.206 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:22.206 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:22.206 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:22.206 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:22.206 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:22.543 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:22.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:22.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:22.544 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:22.997 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:22.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:55:22.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:22.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:22.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:22.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:22.998 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:22.998 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:22.999 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:23.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34072409 2020-03-11T22:55:23.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34072409 '!=' null 2020-03-11T22:55:23.512 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:23.513 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:23.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:23.513 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:23.603 INFO:tasks.workunit.client.0.smithi142.stdout:{ 2020-03-11T22:55:23.603 INFO:tasks.workunit.client.0.smithi142.stdout: "local_reservations": { 2020-03-11T22:55:23.603 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:23.603 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:23.604 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [ 2020-03-11T22:55:23.604 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:23.604 INFO:tasks.workunit.client.0.smithi142.stdout: "priority": 110, 2020-03-11T22:55:23.604 INFO:tasks.workunit.client.0.smithi142.stdout: "items": [ 2020-03-11T22:55:23.604 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:23.604 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "1.0", 2020-03-11T22:55:23.605 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 110, 2020-03-11T22:55:23.605 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:23.605 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:23.605 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:23.605 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:23.605 INFO:tasks.workunit.client.0.smithi142.stdout: ], 2020-03-11T22:55:23.605 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [ 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: { 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: "item": "3.0", 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: "prio": 151, 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: "can_preempt": true 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: ] 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: }, 2020-03-11T22:55:23.606 INFO:tasks.workunit.client.0.smithi142.stdout: "remote_reservations": { 2020-03-11T22:55:23.607 INFO:tasks.workunit.client.0.smithi142.stdout: "max_allowed": 1, 2020-03-11T22:55:23.607 INFO:tasks.workunit.client.0.smithi142.stdout: "min_priority": 0, 2020-03-11T22:55:23.607 INFO:tasks.workunit.client.0.smithi142.stdout: "queues": [], 2020-03-11T22:55:23.607 INFO:tasks.workunit.client.0.smithi142.stdout: "in_progress": [] 2020-03-11T22:55:23.607 INFO:tasks.workunit.client.0.smithi142.stdout: } 2020-03-11T22:55:23.607 INFO:tasks.workunit.client.0.smithi142.stdout:} 2020-03-11T22:55:23.608 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:55:23.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:55:23.712 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:55:23.712 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:55:23.712 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:55:23.712 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:55:23.713 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:55:23.713 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:55:23.713 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:55:24.047 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:55:24.047 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:55:24.051 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:55:24.052 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:55:24.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:55:24.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:55:24.505 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:55:24.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:55:24.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:55:24.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:55:24.506 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:55:24.507 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:55:24.507 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:55:25.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34072409 2020-03-11T22:55:25.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34072409 '!=' null 2020-03-11T22:55:25.002 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:55:25.003 INFO:tasks.workunit.client.0.smithi142.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:25.003 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:55:25.003 INFO:tasks.workunit.client.0.smithi142.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.55987/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:55:25.022 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:25.022 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2020-03-11T22:55:25.023 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2020-03-11T22:55:25.023 DEBUG:teuthology.exit:Finished running handlers